You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Alexey Varlamov <al...@gmail.com> on 2008/01/09 14:07:40 UTC
[build] managing external dependencies
Folks,
We did a few iterations refactoring the subject - let's have one more access ;).
Why? There was a desire to have the following:
1) More fine-grained control over dependencies to empower modularity;
2) Unified approach and shared scripts across Harmony modules - VMs,
classlib, jdktools;
3) Reduce duplication of resources (traffic, space, etc) between
modules within the same workspace;
I'd also add 4) If possible, reuse resources in different workspaces.
This would be quite handy for committing purposes and for
multi-platform development.
Geir once introduced "common_resources" module, which IMO was a step
in the right direction, but it was not fully adapted thus not used
properly. Now the new build for DRLVM is a good stimulus and
opportunity to accomplish the task.
I suggest this time we deprive the classlib of it's state of Harmony
umbilicus and make the "common_resources" a "primordial" module ;)
The idea is to move all shared properties, definitions, tasks and
files to the single location. Besides, the same module should be used
as a central repository for downloading and storing external files.
So, each module would "request" needed resources via standard
facilities (ant tasks), automatically reusing duplicates. And the same
"common_resources" instance can be imported to several workspaces and
even shared between developers.
How?
1) Roughly, the following files need to be moved from classlib to
common_resources:
make/properties.xml
make/depends.properties # ? move only shared items to
simplify maintenance
depends/** except depends/files/ # ? there are several libs/zips +
make definitions being shared
Closer comparison of other make/build-*.xml ant scripts in classlib
and jdktools may give more candidates.
2) Implement tasks or macros for fetching resources, storing them and
providing uniform access: common_resources/download.xml svn.xml etc
3) Refactor make/depends.xml in respective modules to resolve bullet #1.
Surely the plan is very sketchy, we need to agree on general direction first.
Waiting for your comments/objections impatiently :)
--
Alexey
Re: [build] managing external dependencies
Posted by Sian January <si...@googlemail.com>.
Hi Mark,
I've just tried your patch on Windows and it works as you described.
Regards,
Sian
On 18/01/2008, Mark Hindess <ma...@googlemail.com> wrote:
>
>
> Could someone with windows who has run a build since r612947 but before
> r613291 please test the patch attached to:
>
> https://issues.apache.org/jira/browse/HARMONY-5405
>
> I'd like to check that it correctly forces people to get the updated
> zip file (containing a .lib file) and that the rest of the cleanup
> in the patch[0] doesn't break the build.
>
> Thanks,
> Mark.
>
> [0] There were two separate sections in the build-native.xml copying
> the icu link-against libraries - one for z/os .x files and one for
> windows .lib files. The .lib file was also copied in to the hdk on
> linux which was a little bit pointless.
>
> On 17 January 2008 at 19:44, Mark Hindess <ma...@googlemail.com>
> wrote:
> >
> > I've committed r612947 with the trickiest part of the icu dll
> > restructuring now. Hopefully I've not broken too much.
> >
> > A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> > 265M. Of course, some of it will be added back by fetch-depends but
> > even so that is a massive improvement.
> >
> > Thanks to Tim for prompting me to do this - though typically it turned
> > out to be a can of worms[0].
> >
> > Regards,
> > Mark.
> >
> > [0] http://www.answers.com/topic/can-of-worms
> >
> > On 17 January 2008 at 11:31, "Alexey Petrenko"
> > <al...@gmail.com> wrote:
> > > Thanks, Mark!
> > >
> > > SY, Alexey
> > >
> > > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > > >
> > > > On 16 January 2008 at 15:53, "Alexey Petrenko" <
> alexey.a.petrenko@gmail.c
> > om
> > > > wrote:
> > > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > > >
> > > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <
> alexey.v.varlamov@gm
> > ai
> > > l.co
> > > > > m> wrote:
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > We did a few iterations refactoring the subject - let's have
> one
> > mo
> > > re a
> > > > > ccess
> > > > > > > > ;).
> > > > > > > > Why? There was a desire to have the following:
> > > > > > > > 1) More fine-grained control over dependencies to empower
> modular
> > it
> > > y;
> > > > > > > > 2) Unified approach and shared scripts across Harmony
> modules - V
> > Ms
> > > ,
> > > > > > > > classlib, jdktools;
> > > > > > > > 3) Reduce duplication of resources (traffic, space, etc)
> between
> > > > > > > > modules within the same workspace;
> > > > > > > > I'd also add 4) If possible, reuse resources in different
> workspa
> > ce
> > > s.
> > > > > > > > This would be quite handy for committing purposes and for
> > > > > > > > multi-platform development.
> > > > > > >
> > > > > > > I like all of the above but would add Tim's suggestion from a
> few w
> > ee
> > > ks
> > > > > > > back which I think was something like:
> > > > > > >
> > > > > > > 5) Move binary dependencies - like all the icu libs - from the
> enha
> > nc
> > > ed
> > > > > > > to the standard tree in svn and download them via http as we
> do oth
> > er
> > > > > > > dependencies during the fetch-depends stage.
> > > > > >
> > > > > > +1. Checking out tens of megabytes of unrelated binaries is
> really an
> > no
> > > ying
> > > > > !
> > > > > +1
> > > >
> > > > I started looking at fixing this earlier today ... making steady
> > > > progress but I've just got a new laptop so I am a little distracted
> now.
> > > >
> > > > -Mark.
> > > >
> > > >
> > > >
> > >
> >
>
>
>
--
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [build] managing external dependencies
Posted by Tim Ellison <t....@gmail.com>.
Mark Hindess wrote:
> On 21 January 2008 at 12:20, Tim Ellison <t....@gmail.com> wrote:
>> I just updated to r613866 (from being a bit behind doing the merge work)
>> and when I ran fetch-depends it failed in the way Stepan described:
>>
>> File depends/libs/windows.x86/icu-3.4/icu-3.4.zip has incorrect md5
>> checksum.
>> Expected: 4ab256e309d1ceb9779b10cf32c79dab (or )
>> Found: f8b46d05e08ca98e42738f159ae1dd40
>>
>> So I deleted the depends/libs/windows.x86/icu-3.4 directory and fetched
>> the dependencies again (and it worked).
>>
>> Regards,
>> Tim
>
> Damn. I think you only get the warning that tells you what to do if you
> run "check-depends".
>
> The bad news is I need to break it again ... this time for bcprov in
> order to move the download to a non-bouncycastle.org server as they
> requested last year. A similar fix will apply; removing the
> depends/jars/bcprov-jdk15-138 directory.
Consider a new thread, clearly labeled, to let people know how to fix
their workspace. Not everyone will be reading the details of managing
external dependencies.
Regards,
Tim
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 21 January 2008 at 12:20, Tim Ellison <t....@gmail.com> wrote:
> I just updated to r613866 (from being a bit behind doing the merge work)
> and when I ran fetch-depends it failed in the way Stepan described:
>
> File depends/libs/windows.x86/icu-3.4/icu-3.4.zip has incorrect md5
> checksum.
> Expected: 4ab256e309d1ceb9779b10cf32c79dab (or )
> Found: f8b46d05e08ca98e42738f159ae1dd40
>
> So I deleted the depends/libs/windows.x86/icu-3.4 directory and fetched
> the dependencies again (and it worked).
>
> Regards,
> Tim
Damn. I think you only get the warning that tells you what to do if you
run "check-depends".
The bad news is I need to break it again ... this time for bcprov in
order to move the download to a non-bouncycastle.org server as they
requested last year. A similar fix will apply; removing the
depends/jars/bcprov-jdk15-138 directory.
Regards,
Mark.
> Mark Hindess wrote:
> > Could someone with windows who has run a build since r612947 but before
> > r613291 please test the patch attached to:
> >
> > https://issues.apache.org/jira/browse/HARMONY-5405
> >
> > I'd like to check that it correctly forces people to get the updated
> > zip file (containing a .lib file) and that the rest of the cleanup
> > in the patch[0] doesn't break the build.
> >
> > Thanks,
> > Mark.
> >
> > [0] There were two separate sections in the build-native.xml copying
> > the icu link-against libraries - one for z/os .x files and one for
> > windows .lib files. The .lib file was also copied in to the hdk on
> > linux which was a little bit pointless.
> >
> > On 17 January 2008 at 19:44, Mark Hindess <ma...@googlemail.com> wro
> te:
> >> I've committed r612947 with the trickiest part of the icu dll
> >> restructuring now. Hopefully I've not broken too much.
> >>
> >> A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> >> 265M. Of course, some of it will be added back by fetch-depends but
> >> even so that is a massive improvement.
> >>
> >> Thanks to Tim for prompting me to do this - though typically it turned
> >> out to be a can of worms[0].
> >>
> >> Regards,
> >> Mark.
> >>
> >> [0] http://www.answers.com/topic/can-of-worms
> >>
> >> On 17 January 2008 at 11:31, "Alexey Petrenko"
> >> <al...@gmail.com> wrote:
> >>> Thanks, Mark!
> >>>
> >>> SY, Alexey
> >>>
> >>> 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> >>>> On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.
> c
> >> om
> >>>> wrote:
> >>>>> 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> >>>>>> 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> >>>>>>> On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gm
> >> ai
> >>> l.co
> >>>>> m> wrote:
> >>>>>>>> Folks,
> >>>>>>>>
> >>>>>>>> We did a few iterations refactoring the subject - let's have one
> >> mo
> >>> re a
> >>>>> ccess
> >>>>>>>> ;).
> >>>>>>>> Why? There was a desire to have the following:
> >>>>>>>> 1) More fine-grained control over dependencies to empower modular
> >> it
> >>> y;
> >>>>>>>> 2) Unified approach and shared scripts across Harmony modules - V
> >> Ms
> >>> ,
> >>>>>>>> classlib, jdktools;
> >>>>>>>> 3) Reduce duplication of resources (traffic, space, etc) between
> >>>>>>>> modules within the same workspace;
> >>>>>>>> I'd also add 4) If possible, reuse resources in different workspa
> >> ce
> >>> s.
> >>>>>>>> This would be quite handy for committing purposes and for
> >>>>>>>> multi-platform development.
> >>>>>>> I like all of the above but would add Tim's suggestion from a few w
> >> ee
> >>> ks
> >>>>>>> back which I think was something like:
> >>>>>>>
> >>>>>>> 5) Move binary dependencies - like all the icu libs - from the enha
> >> nc
> >>> ed
> >>>>>>> to the standard tree in svn and download them via http as we do oth
> >> er
> >>>>>>> dependencies during the fetch-depends stage.
> >>>>>> +1. Checking out tens of megabytes of unrelated binaries is really an
> >> no
> >>> ying
> >>>>> !
> >>>>> +1
> >>>> I started looking at fixing this earlier today ... making steady
> >>>> progress but I've just got a new laptop so I am a little distracted now.
> >>>>
> >>>> -Mark.
> >>>>
> >>>>
> >>>>
> >
> >
> >
>
Re: [build] managing external dependencies
Posted by Tim Ellison <t....@gmail.com>.
I just updated to r613866 (from being a bit behind doing the merge work)
and when I ran fetch-depends it failed in the way Stepan described:
File depends/libs/windows.x86/icu-3.4/icu-3.4.zip has incorrect md5
checksum.
Expected: 4ab256e309d1ceb9779b10cf32c79dab (or )
Found: f8b46d05e08ca98e42738f159ae1dd40
So I deleted the depends/libs/windows.x86/icu-3.4 directory and fetched
the dependencies again (and it worked).
Regards,
Tim
Mark Hindess wrote:
> Could someone with windows who has run a build since r612947 but before
> r613291 please test the patch attached to:
>
> https://issues.apache.org/jira/browse/HARMONY-5405
>
> I'd like to check that it correctly forces people to get the updated
> zip file (containing a .lib file) and that the rest of the cleanup
> in the patch[0] doesn't break the build.
>
> Thanks,
> Mark.
>
> [0] There were two separate sections in the build-native.xml copying
> the icu link-against libraries - one for z/os .x files and one for
> windows .lib files. The .lib file was also copied in to the hdk on
> linux which was a little bit pointless.
>
> On 17 January 2008 at 19:44, Mark Hindess <ma...@googlemail.com> wrote:
>> I've committed r612947 with the trickiest part of the icu dll
>> restructuring now. Hopefully I've not broken too much.
>>
>> A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
>> 265M. Of course, some of it will be added back by fetch-depends but
>> even so that is a massive improvement.
>>
>> Thanks to Tim for prompting me to do this - though typically it turned
>> out to be a can of worms[0].
>>
>> Regards,
>> Mark.
>>
>> [0] http://www.answers.com/topic/can-of-worms
>>
>> On 17 January 2008 at 11:31, "Alexey Petrenko"
>> <al...@gmail.com> wrote:
>>> Thanks, Mark!
>>>
>>> SY, Alexey
>>>
>>> 2008/1/16, Mark Hindess <ma...@googlemail.com>:
>>>> On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.c
>> om
>>>> wrote:
>>>>> 2008/1/16, Alexey Varlamov <al...@gmail.com>:
>>>>>> 2008/1/9, Mark Hindess <ma...@googlemail.com>:
>>>>>>> On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gm
>> ai
>>> l.co
>>>>> m> wrote:
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> We did a few iterations refactoring the subject - let's have one
>> mo
>>> re a
>>>>> ccess
>>>>>>>> ;).
>>>>>>>> Why? There was a desire to have the following:
>>>>>>>> 1) More fine-grained control over dependencies to empower modular
>> it
>>> y;
>>>>>>>> 2) Unified approach and shared scripts across Harmony modules - V
>> Ms
>>> ,
>>>>>>>> classlib, jdktools;
>>>>>>>> 3) Reduce duplication of resources (traffic, space, etc) between
>>>>>>>> modules within the same workspace;
>>>>>>>> I'd also add 4) If possible, reuse resources in different workspa
>> ce
>>> s.
>>>>>>>> This would be quite handy for committing purposes and for
>>>>>>>> multi-platform development.
>>>>>>> I like all of the above but would add Tim's suggestion from a few w
>> ee
>>> ks
>>>>>>> back which I think was something like:
>>>>>>>
>>>>>>> 5) Move binary dependencies - like all the icu libs - from the enha
>> nc
>>> ed
>>>>>>> to the standard tree in svn and download them via http as we do oth
>> er
>>>>>>> dependencies during the fetch-depends stage.
>>>>>> +1. Checking out tens of megabytes of unrelated binaries is really an
>> no
>>> ying
>>>>> !
>>>>> +1
>>>> I started looking at fixing this earlier today ... making steady
>>>> progress but I've just got a new laptop so I am a little distracted now.
>>>>
>>>> -Mark.
>>>>
>>>>
>>>>
>
>
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
Could someone with windows who has run a build since r612947 but before
r613291 please test the patch attached to:
https://issues.apache.org/jira/browse/HARMONY-5405
I'd like to check that it correctly forces people to get the updated
zip file (containing a .lib file) and that the rest of the cleanup
in the patch[0] doesn't break the build.
Thanks,
Mark.
[0] There were two separate sections in the build-native.xml copying
the icu link-against libraries - one for z/os .x files and one for
windows .lib files. The .lib file was also copied in to the hdk on
linux which was a little bit pointless.
On 17 January 2008 at 19:44, Mark Hindess <ma...@googlemail.com> wrote:
>
> I've committed r612947 with the trickiest part of the icu dll
> restructuring now. Hopefully I've not broken too much.
>
> A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> 265M. Of course, some of it will be added back by fetch-depends but
> even so that is a massive improvement.
>
> Thanks to Tim for prompting me to do this - though typically it turned
> out to be a can of worms[0].
>
> Regards,
> Mark.
>
> [0] http://www.answers.com/topic/can-of-worms
>
> On 17 January 2008 at 11:31, "Alexey Petrenko"
> <al...@gmail.com> wrote:
> > Thanks, Mark!
> >
> > SY, Alexey
> >
> > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > >
> > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.c
> om
> > > wrote:
> > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > >
> > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gm
> ai
> > l.co
> > > > m> wrote:
> > > > > > > Folks,
> > > > > > >
> > > > > > > We did a few iterations refactoring the subject - let's have one
> mo
> > re a
> > > > ccess
> > > > > > > ;).
> > > > > > > Why? There was a desire to have the following:
> > > > > > > 1) More fine-grained control over dependencies to empower modular
> it
> > y;
> > > > > > > 2) Unified approach and shared scripts across Harmony modules - V
> Ms
> > ,
> > > > > > > classlib, jdktools;
> > > > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > > > modules within the same workspace;
> > > > > > > I'd also add 4) If possible, reuse resources in different workspa
> ce
> > s.
> > > > > > > This would be quite handy for committing purposes and for
> > > > > > > multi-platform development.
> > > > > >
> > > > > > I like all of the above but would add Tim's suggestion from a few w
> ee
> > ks
> > > > > > back which I think was something like:
> > > > > >
> > > > > > 5) Move binary dependencies - like all the icu libs - from the enha
> nc
> > ed
> > > > > > to the standard tree in svn and download them via http as we do oth
> er
> > > > > > dependencies during the fetch-depends stage.
> > > > >
> > > > > +1. Checking out tens of megabytes of unrelated binaries is really an
> no
> > ying
> > > > !
> > > > +1
> > >
> > > I started looking at fixing this earlier today ... making steady
> > > progress but I've just got a new laptop so I am a little distracted now.
> > >
> > > -Mark.
> > >
> > >
> > >
> >
>
Re: [build] managing external dependencies
Posted by Alexey Varlamov <al...@gmail.com>.
2008/1/18, Mark Hindess <ma...@googlemail.com>:
>
> I've committed r612947 with the trickiest part of the icu dll
> restructuring now. Hopefully I've not broken too much.
>
> A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> 265M. Of course, some of it will be added back by fetch-depends but
> even so that is a massive improvement.
Well done, thank you!
>
> Thanks to Tim for prompting me to do this - though typically it turned
> out to be a can of worms[0].
There is no evil without good, and
Well begun is half done ;)
Actually, I'm also experiencing some troubles implementing the subject
proposal, stumbled at trying to fit everything and everyone :(
Finally decided to pilot the drlvm first, then see how to approach
jdktools and classlib.
--
Alexey
>
> Regards,
> Mark.
>
> [0] http://www.answers.com/topic/can-of-worms
>
> On 17 January 2008 at 11:31, "Alexey Petrenko"
> <al...@gmail.com> wrote:
> > Thanks, Mark!
> >
> > SY, Alexey
> >
> > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > >
> > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> > > wrote:
> > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > >
> > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmai
> > l.co
> > > > m> wrote:
> > > > > > > Folks,
> > > > > > >
> > > > > > > We did a few iterations refactoring the subject - let's have one mo
> > re a
> > > > ccess
> > > > > > > ;).
> > > > > > > Why? There was a desire to have the following:
> > > > > > > 1) More fine-grained control over dependencies to empower modularit
> > y;
> > > > > > > 2) Unified approach and shared scripts across Harmony modules - VMs
> > ,
> > > > > > > classlib, jdktools;
> > > > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > > > modules within the same workspace;
> > > > > > > I'd also add 4) If possible, reuse resources in different workspace
> > s.
> > > > > > > This would be quite handy for committing purposes and for
> > > > > > > multi-platform development.
> > > > > >
> > > > > > I like all of the above but would add Tim's suggestion from a few wee
> > ks
> > > > > > back which I think was something like:
> > > > > >
> > > > > > 5) Move binary dependencies - like all the icu libs - from the enhanc
> > ed
> > > > > > to the standard tree in svn and download them via http as we do other
> > > > > > dependencies during the fetch-depends stage.
> > > > >
> > > > > +1. Checking out tens of megabytes of unrelated binaries is really anno
> > ying
> > > > !
> > > > +1
> > >
> > > I started looking at fixing this earlier today ... making steady
> > > progress but I've just got a new laptop so I am a little distracted now.
> > >
> > > -Mark.
> > >
> > >
> > >
> >
>
>
>
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
Yep, it works fine for me after your last commit.
Thanks.
SY, Alexey
2008/1/23, Mark Hindess <ma...@googlemail.com>:
>
> On 23 January 2008 at 2:32, "Alexey Petrenko" <al...@gmail.com>
> wrote:
> > 2008/1/22, Mark Hindess <ma...@googlemail.com>:
> > >
> > > On 22 January 2008 at 23:07, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> > >
> > > wrote:
> > > > Looks like the problem with libstdc++ versions again. drlvm build
> > > > creates symbolic links to icu libraries in linux.x86 while now this
> > > > directory is empty and the files exists in linux.x86.libstdc++6 only.
> > > I don't see how you'd get this type of issue if -Duse.libstdc++6=true
> > > (or =false) was passed consistently to both the classlib and drlvm build.
> > Hmm... It works if build vm separately. But does not work while federated
> > build.
>
> I set "use.libstdc++6=true" in ~/.harmony-classlib.properties which gets
> sourced (but both classlib and drlvm since drlvm uses classlibs
> properties.xml). This meant I wasn't seeing the fact that use.libstdc++6
> was not being passed correctly from the federation build down to classlib
> and drlvm.
>
> I hope that r614462 should fix this for you.
>
> -Mark
>
>
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 23 January 2008 at 2:32, "Alexey Petrenko" <al...@gmail.com>
wrote:
> 2008/1/22, Mark Hindess <ma...@googlemail.com>:
> >
> > On 22 January 2008 at 23:07, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> >
> > wrote:
> > > Looks like the problem with libstdc++ versions again. drlvm build
> > > creates symbolic links to icu libraries in linux.x86 while now this
> > > directory is empty and the files exists in linux.x86.libstdc++6 only.
> > I don't see how you'd get this type of issue if -Duse.libstdc++6=true
> > (or =false) was passed consistently to both the classlib and drlvm build.
> Hmm... It works if build vm separately. But does not work while federated
> build.
Odd. I always use the federated build and it is working for me. I'll
take another look.
-Mark.
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 23 January 2008 at 2:32, "Alexey Petrenko" <al...@gmail.com>
wrote:
> 2008/1/22, Mark Hindess <ma...@googlemail.com>:
> >
> > On 22 January 2008 at 23:07, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> >
> > wrote:
> > > Looks like the problem with libstdc++ versions again. drlvm build
> > > creates symbolic links to icu libraries in linux.x86 while now this
> > > directory is empty and the files exists in linux.x86.libstdc++6 only.
> > I don't see how you'd get this type of issue if -Duse.libstdc++6=true
> > (or =false) was passed consistently to both the classlib and drlvm build.
> Hmm... It works if build vm separately. But does not work while federated
> build.
I set "use.libstdc++6=true" in ~/.harmony-classlib.properties which gets
sourced (but both classlib and drlvm since drlvm uses classlibs
properties.xml). This meant I wasn't seeing the fact that use.libstdc++6
was not being passed correctly from the federation build down to classlib
and drlvm.
I hope that r614462 should fix this for you.
-Mark
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
2008/1/22, Mark Hindess <ma...@googlemail.com>:
>
> On 22 January 2008 at 23:07, "Alexey Petrenko" <al...@gmail.com>
> wrote:
> > Looks like the problem with libstdc++ versions again. drlvm build
> > creates symbolic links to icu libraries in linux.x86 while now this
> > directory is empty and the files exists in linux.x86.libstdc++6 only.
> I don't see how you'd get this type of issue if -Duse.libstdc++6=true
> (or =false) was passed consistently to both the classlib and drlvm build.
Hmm... It works if build vm separately. But does not work while federated build.
SY, Alexey
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 22 January 2008 at 23:07, "Alexey Petrenko" <al...@gmail.com>
wrote:
> Looks like the problem with libstdc++ versions again. drlvm build
> creates symbolic links to icu libraries in linux.x86 while now this
> directory is empty and the files exists in linux.x86.libstdc++6 only.
I don't see how you'd get this type of issue if -Duse.libstdc++6=true
(or =false) was passed consistently to both the classlib and drlvm build.
> I could not find an option in drlvm build to fix the situation. Any ideas?
>
> BTW, I'm curios how it worked before when drlvm linked v5 icu on v6 machines.
> ..
It was broken before - i.e. it would ignore the libstdc++6 setting and
link against the libstdc++5 versions which happened to have the same
interface so didn't cause any problems - but before all versions existed
because the directory tree was checked in for all platforms/variants.
-Mark.
> 2008/1/22, Alexey Petrenko <al...@gmail.com>:
> > I got the following error on federated build which looks as a result
> > of this change:
> >
> > === cut ===
> > [exec] -init-unix:
> > [exec] [symlink] ln -s
> > /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4
> /libicuuc.so.34
> > /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4
> /libicuuc.so
> > [exec] [symlink] /bin/ln: creating symbolic link
> > `/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.
> 4/libicuuc.so':
> > File exists
> >
> > [exec] BUILD FAILED
> > [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:600
> :
> > The following error occurred while executing this line:
> > [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:607
> :
> > The following error occurred while executing this line:
> > [exec] /localdisk/aapetren/harmony/working_vm/build/make/build_compone
> nt.xml:74:
> > The following error occurred while executing this line:
> > [exec] /localdisk/aapetren/harmony/working_vm/build/lnx_ia32_gcc_relea
> se/semis/build/init_vm.vmcore.xml:23:
> > ln failed with return code 1
> >
> > [exec] Total time: 2 minutes 36 seconds
> > [exec] *
> > [exec] * Please, refer to README.txt for details.
> >
> > BUILD FAILED
> > /localdisk/aapetren/harmony/build.xml:407: exec returned: 1
> >
> > Total time: 7 minutes 45 seconds
> > === cut ===
> >
> > Will try to find a solution.
> > Any suggestions are welcome :)
> >
> > SY, Alexey
> >
> > 2008/1/17, Mark Hindess <ma...@googlemail.com>:
> > >
> > > I've committed r612947 with the trickiest part of the icu dll
> > > restructuring now. Hopefully I've not broken too much.
> > >
> > > A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> > > 265M. Of course, some of it will be added back by fetch-depends but
> > > even so that is a massive improvement.
> > >
> > > Thanks to Tim for prompting me to do this - though typically it turned
> > > out to be a can of worms[0].
> > >
> > > Regards,
> > > Mark.
> > >
> > > [0] http://www.answers.com/topic/can-of-worms
> > >
> > > On 17 January 2008 at 11:31, "Alexey Petrenko"
> > > <al...@gmail.com> wrote:
> > > > Thanks, Mark!
> > > >
> > > > SY, Alexey
> > > >
> > > > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > > > >
> > > > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gma
> il.com
> > > > > wrote:
> > > > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > > > >
> > > > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamo
> v@gmai
> > > > l.co
> > > > > > m> wrote:
> > > > > > > > > Folks,
> > > > > > > > >
> > > > > > > > > We did a few iterations refactoring the subject - let's have
> one mo
> > > > re a
> > > > > > ccess
> > > > > > > > > ;).
> > > > > > > > > Why? There was a desire to have the following:
> > > > > > > > > 1) More fine-grained control over dependencies to empower mod
> ularit
> > > > y;
> > > > > > > > > 2) Unified approach and shared scripts across Harmony modules
> - VMs
> > > > ,
> > > > > > > > > classlib, jdktools;
> > > > > > > > > 3) Reduce duplication of resources (traffic, space, etc) betw
> een
> > > > > > > > > modules within the same workspace;
> > > > > > > > > I'd also add 4) If possible, reuse resources in different wor
> kspace
> > > > s.
> > > > > > > > > This would be quite handy for committing purposes and for
> > > > > > > > > multi-platform development.
> > > > > > > >
> > > > > > > > I like all of the above but would add Tim's suggestion from a f
> ew wee
> > > > ks
> > > > > > > > back which I think was something like:
> > > > > > > >
> > > > > > > > 5) Move binary dependencies - like all the icu libs - from the
> enhanc
> > > > ed
> > > > > > > > to the standard tree in svn and download them via http as we do
> other
> > > > > > > > dependencies during the fetch-depends stage.
> > > > > > >
> > > > > > > +1. Checking out tens of megabytes of unrelated binaries is reall
> y anno
> > > > ying
> > > > > > !
> > > > > > +1
> > > > >
> > > > > I started looking at fixing this earlier today ... making steady
> > > > > progress but I've just got a new laptop so I am a little distracted n
> ow.
> > > > >
> > > > > -Mark.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
Looks like the problem with libstdc++ versions again. drlvm build
creates symbolic links to icu libraries in linux.x86 while now this
directory is empty and the files exists in linux.x86.libstdc++6 only.
I could not find an option in drlvm build to fix the situation. Any ideas?
BTW, I'm curios how it worked before when drlvm linked v5 icu on v6 machines...
SY, Alexey
2008/1/22, Alexey Petrenko <al...@gmail.com>:
> I got the following error on federated build which looks as a result
> of this change:
>
> === cut ===
> [exec] -init-unix:
> [exec] [symlink] ln -s
> /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so.34
> /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so
> [exec] [symlink] /bin/ln: creating symbolic link
> `/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so':
> File exists
>
> [exec] BUILD FAILED
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:600:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:607:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build_component.xml:74:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/lnx_ia32_gcc_release/semis/build/init_vm.vmcore.xml:23:
> ln failed with return code 1
>
> [exec] Total time: 2 minutes 36 seconds
> [exec] *
> [exec] * Please, refer to README.txt for details.
>
> BUILD FAILED
> /localdisk/aapetren/harmony/build.xml:407: exec returned: 1
>
> Total time: 7 minutes 45 seconds
> === cut ===
>
> Will try to find a solution.
> Any suggestions are welcome :)
>
> SY, Alexey
>
> 2008/1/17, Mark Hindess <ma...@googlemail.com>:
> >
> > I've committed r612947 with the trickiest part of the icu dll
> > restructuring now. Hopefully I've not broken too much.
> >
> > A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> > 265M. Of course, some of it will be added back by fetch-depends but
> > even so that is a massive improvement.
> >
> > Thanks to Tim for prompting me to do this - though typically it turned
> > out to be a can of worms[0].
> >
> > Regards,
> > Mark.
> >
> > [0] http://www.answers.com/topic/can-of-worms
> >
> > On 17 January 2008 at 11:31, "Alexey Petrenko"
> > <al...@gmail.com> wrote:
> > > Thanks, Mark!
> > >
> > > SY, Alexey
> > >
> > > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > > >
> > > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> > > > wrote:
> > > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > > >
> > > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmai
> > > l.co
> > > > > m> wrote:
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > We did a few iterations refactoring the subject - let's have one mo
> > > re a
> > > > > ccess
> > > > > > > > ;).
> > > > > > > > Why? There was a desire to have the following:
> > > > > > > > 1) More fine-grained control over dependencies to empower modularit
> > > y;
> > > > > > > > 2) Unified approach and shared scripts across Harmony modules - VMs
> > > ,
> > > > > > > > classlib, jdktools;
> > > > > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > > > > modules within the same workspace;
> > > > > > > > I'd also add 4) If possible, reuse resources in different workspace
> > > s.
> > > > > > > > This would be quite handy for committing purposes and for
> > > > > > > > multi-platform development.
> > > > > > >
> > > > > > > I like all of the above but would add Tim's suggestion from a few wee
> > > ks
> > > > > > > back which I think was something like:
> > > > > > >
> > > > > > > 5) Move binary dependencies - like all the icu libs - from the enhanc
> > > ed
> > > > > > > to the standard tree in svn and download them via http as we do other
> > > > > > > dependencies during the fetch-depends stage.
> > > > > >
> > > > > > +1. Checking out tens of megabytes of unrelated binaries is really anno
> > > ying
> > > > > !
> > > > > +1
> > > >
> > > > I started looking at fixing this earlier today ... making steady
> > > > progress but I've just got a new laptop so I am a little distracted now.
> > > >
> > > > -Mark.
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 22 January 2008 at 22:32, "Alexey Petrenko" <al...@gmail.com>
wrote:
> I got the following error on federated build which looks as a result
> of this change:
Can you explain why you think that this is the result of my recent changes?
Nothing I did creates the file (actually a symlink I imagine) that the
error message claims exists. The only thing I can find that creates it
is the line in the vmcore.xml build file.
To be honest, this has puzzled me for a while and needs fixing for
several reasons:
1) I added a delete to try to make sure the target of the symlink
was missing (I did this at least a week ago not as part of the
recent changes.)
2) the symlink task has overwrite="yes" which I'd imagine would
force the overwriting of the existing symlink but then it seems
to be executing "ln -s" rather than "ln -sf" as one might expect.
3) drlvm build shouldn't really be modifying classlib files like
this anyway - and if it does it should really remove anything
created as part of the drlvm clean target.
4) Despite my attempts to patch over this problem (see 1) I still
get problems with packaging for debian because of this error.
I haven't bothered to look at it further because I was hoping these
files would be moved to common_resources (since they aren't needed
by classlib any more only by drlvm) and that the problem would
be removed in the process.
I was making this batch of icu dependency improvements because I think
it should make it easier to move them to common_resources and
because it brings an immediate benefit in terms of checkout size/time.
Regards,
-Mark
> === cut ===
> [exec] -init-unix:
> [exec] [symlink] ln -s
> /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/l
> ibicuuc.so.34
> /localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/l
> ibicuuc.so
> [exec] [symlink] /bin/ln: creating symbolic link
> `/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/
> libicuuc.so':
> File exists
>
> [exec] BUILD FAILED
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:600:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:607:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/make/build_component
> .xml:74:
> The following error occurred while executing this line:
> [exec] /localdisk/aapetren/harmony/working_vm/build/lnx_ia32_gcc_release
> /semis/build/init_vm.vmcore.xml:23:
> ln failed with return code 1
>
> [exec] Total time: 2 minutes 36 seconds
> [exec] *
> [exec] * Please, refer to README.txt for details.
>
> BUILD FAILED
> /localdisk/aapetren/harmony/build.xml:407: exec returned: 1
>
> Total time: 7 minutes 45 seconds
> === cut ===
>
> Will try to find a solution.
> Any suggestions are welcome :)
>
> SY, Alexey
>
> 2008/1/17, Mark Hindess <ma...@googlemail.com>:
> >
> > I've committed r612947 with the trickiest part of the icu dll
> > restructuring now. Hopefully I've not broken too much.
> >
> > A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> > 265M. Of course, some of it will be added back by fetch-depends but
> > even so that is a massive improvement.
> >
> > Thanks to Tim for prompting me to do this - though typically it turned
> > out to be a can of worms[0].
> >
> > Regards,
> > Mark.
> >
> > [0] http://www.answers.com/topic/can-of-worms
> >
> > On 17 January 2008 at 11:31, "Alexey Petrenko"
> > <al...@gmail.com> wrote:
> > > Thanks, Mark!
> > >
> > > SY, Alexey
> > >
> > > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > > >
> > > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail
> .com
> > > > wrote:
> > > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > > >
> > > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@
> gmai
> > > l.co
> > > > > m> wrote:
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > We did a few iterations refactoring the subject - let's have on
> e mo
> > > re a
> > > > > ccess
> > > > > > > > ;).
> > > > > > > > Why? There was a desire to have the following:
> > > > > > > > 1) More fine-grained control over dependencies to empower modul
> arit
> > > y;
> > > > > > > > 2) Unified approach and shared scripts across Harmony modules -
> VMs
> > > ,
> > > > > > > > classlib, jdktools;
> > > > > > > > 3) Reduce duplication of resources (traffic, space, etc) betwee
> n
> > > > > > > > modules within the same workspace;
> > > > > > > > I'd also add 4) If possible, reuse resources in different works
> pace
> > > s.
> > > > > > > > This would be quite handy for committing purposes and for
> > > > > > > > multi-platform development.
> > > > > > >
> > > > > > > I like all of the above but would add Tim's suggestion from a few
> wee
> > > ks
> > > > > > > back which I think was something like:
> > > > > > >
> > > > > > > 5) Move binary dependencies - like all the icu libs - from the en
> hanc
> > > ed
> > > > > > > to the standard tree in svn and download them via http as we do o
> ther
> > > > > > > dependencies during the fetch-depends stage.
> > > > > >
> > > > > > +1. Checking out tens of megabytes of unrelated binaries is really
> anno
> > > ying
> > > > > !
> > > > > +1
> > > >
> > > > I started looking at fixing this earlier today ... making steady
> > > > progress but I've just got a new laptop so I am a little distracted now
> .
> > > >
> > > > -Mark.
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
I got the following error on federated build which looks as a result
of this change:
=== cut ===
[exec] -init-unix:
[exec] [symlink] ln -s
/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so.34
/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so
[exec] [symlink] /bin/ln: creating symbolic link
`/localdisk/aapetren/harmony/working_classlib/depends/libs/linux.x86/icu-3.4/libicuuc.so':
File exists
[exec] BUILD FAILED
[exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:600:
The following error occurred while executing this line:
[exec] /localdisk/aapetren/harmony/working_vm/build/make/build.xml:607:
The following error occurred while executing this line:
[exec] /localdisk/aapetren/harmony/working_vm/build/make/build_component.xml:74:
The following error occurred while executing this line:
[exec] /localdisk/aapetren/harmony/working_vm/build/lnx_ia32_gcc_release/semis/build/init_vm.vmcore.xml:23:
ln failed with return code 1
[exec] Total time: 2 minutes 36 seconds
[exec] *
[exec] * Please, refer to README.txt for details.
BUILD FAILED
/localdisk/aapetren/harmony/build.xml:407: exec returned: 1
Total time: 7 minutes 45 seconds
=== cut ===
Will try to find a solution.
Any suggestions are welcome :)
SY, Alexey
2008/1/17, Mark Hindess <ma...@googlemail.com>:
>
> I've committed r612947 with the trickiest part of the icu dll
> restructuring now. Hopefully I've not broken too much.
>
> A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
> 265M. Of course, some of it will be added back by fetch-depends but
> even so that is a massive improvement.
>
> Thanks to Tim for prompting me to do this - though typically it turned
> out to be a can of worms[0].
>
> Regards,
> Mark.
>
> [0] http://www.answers.com/topic/can-of-worms
>
> On 17 January 2008 at 11:31, "Alexey Petrenko"
> <al...@gmail.com> wrote:
> > Thanks, Mark!
> >
> > SY, Alexey
> >
> > 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> > >
> > > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> > > wrote:
> > > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > > >
> > > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmai
> > l.co
> > > > m> wrote:
> > > > > > > Folks,
> > > > > > >
> > > > > > > We did a few iterations refactoring the subject - let's have one mo
> > re a
> > > > ccess
> > > > > > > ;).
> > > > > > > Why? There was a desire to have the following:
> > > > > > > 1) More fine-grained control over dependencies to empower modularit
> > y;
> > > > > > > 2) Unified approach and shared scripts across Harmony modules - VMs
> > ,
> > > > > > > classlib, jdktools;
> > > > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > > > modules within the same workspace;
> > > > > > > I'd also add 4) If possible, reuse resources in different workspace
> > s.
> > > > > > > This would be quite handy for committing purposes and for
> > > > > > > multi-platform development.
> > > > > >
> > > > > > I like all of the above but would add Tim's suggestion from a few wee
> > ks
> > > > > > back which I think was something like:
> > > > > >
> > > > > > 5) Move binary dependencies - like all the icu libs - from the enhanc
> > ed
> > > > > > to the standard tree in svn and download them via http as we do other
> > > > > > dependencies during the fetch-depends stage.
> > > > >
> > > > > +1. Checking out tens of megabytes of unrelated binaries is really anno
> > ying
> > > > !
> > > > +1
> > >
> > > I started looking at fixing this earlier today ... making steady
> > > progress but I've just got a new laptop so I am a little distracted now.
> > >
> > > -Mark.
> > >
> > >
> > >
> >
>
>
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
I've committed r612947 with the trickiest part of the icu dll
restructuring now. Hopefully I've not broken too much.
A clean (pre-fetch-depends) classlib svn tree shrinks from 542M to just
265M. Of course, some of it will be added back by fetch-depends but
even so that is a massive improvement.
Thanks to Tim for prompting me to do this - though typically it turned
out to be a can of worms[0].
Regards,
Mark.
[0] http://www.answers.com/topic/can-of-worms
On 17 January 2008 at 11:31, "Alexey Petrenko"
<al...@gmail.com> wrote:
> Thanks, Mark!
>
> SY, Alexey
>
> 2008/1/16, Mark Hindess <ma...@googlemail.com>:
> >
> > On 16 January 2008 at 15:53, "Alexey Petrenko" <alexey.a.petrenko@gmail.com
> > wrote:
> > > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > > >
> > > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmai
> l.co
> > > m> wrote:
> > > > > > Folks,
> > > > > >
> > > > > > We did a few iterations refactoring the subject - let's have one mo
> re a
> > > ccess
> > > > > > ;).
> > > > > > Why? There was a desire to have the following:
> > > > > > 1) More fine-grained control over dependencies to empower modularit
> y;
> > > > > > 2) Unified approach and shared scripts across Harmony modules - VMs
> ,
> > > > > > classlib, jdktools;
> > > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > > modules within the same workspace;
> > > > > > I'd also add 4) If possible, reuse resources in different workspace
> s.
> > > > > > This would be quite handy for committing purposes and for
> > > > > > multi-platform development.
> > > > >
> > > > > I like all of the above but would add Tim's suggestion from a few wee
> ks
> > > > > back which I think was something like:
> > > > >
> > > > > 5) Move binary dependencies - like all the icu libs - from the enhanc
> ed
> > > > > to the standard tree in svn and download them via http as we do other
> > > > > dependencies during the fetch-depends stage.
> > > >
> > > > +1. Checking out tens of megabytes of unrelated binaries is really anno
> ying
> > > !
> > > +1
> >
> > I started looking at fixing this earlier today ... making steady
> > progress but I've just got a new laptop so I am a little distracted now.
> >
> > -Mark.
> >
> >
> >
>
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
Thanks, Mark!
SY, Alexey
2008/1/16, Mark Hindess <ma...@googlemail.com>:
>
> On 16 January 2008 at 15:53, "Alexey Petrenko" <al...@gmail.com> wrote:
> > 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > > >
> > > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmail.co
> > m> wrote:
> > > > > Folks,
> > > > >
> > > > > We did a few iterations refactoring the subject - let's have one more a
> > ccess
> > > > > ;).
> > > > > Why? There was a desire to have the following:
> > > > > 1) More fine-grained control over dependencies to empower modularity;
> > > > > 2) Unified approach and shared scripts across Harmony modules - VMs,
> > > > > classlib, jdktools;
> > > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > > modules within the same workspace;
> > > > > I'd also add 4) If possible, reuse resources in different workspaces.
> > > > > This would be quite handy for committing purposes and for
> > > > > multi-platform development.
> > > >
> > > > I like all of the above but would add Tim's suggestion from a few weeks
> > > > back which I think was something like:
> > > >
> > > > 5) Move binary dependencies - like all the icu libs - from the enhanced
> > > > to the standard tree in svn and download them via http as we do other
> > > > dependencies during the fetch-depends stage.
> > >
> > > +1. Checking out tens of megabytes of unrelated binaries is really annoying
> > !
> > +1
>
> I started looking at fixing this earlier today ... making steady
> progress but I've just got a new laptop so I am a little distracted now.
>
> -Mark.
>
>
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 16 January 2008 at 15:53, "Alexey Petrenko" <al...@gmail.com> wrote:
> 2008/1/16, Alexey Varlamov <al...@gmail.com>:
> > 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> > >
> > > On 9 January 2008 at 19:07, "Alexey Varlamov" <alexey.v.varlamov@gmail.co
> m> wrote:
> > > > Folks,
> > > >
> > > > We did a few iterations refactoring the subject - let's have one more a
> ccess
> > > > ;).
> > > > Why? There was a desire to have the following:
> > > > 1) More fine-grained control over dependencies to empower modularity;
> > > > 2) Unified approach and shared scripts across Harmony modules - VMs,
> > > > classlib, jdktools;
> > > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > > modules within the same workspace;
> > > > I'd also add 4) If possible, reuse resources in different workspaces.
> > > > This would be quite handy for committing purposes and for
> > > > multi-platform development.
> > >
> > > I like all of the above but would add Tim's suggestion from a few weeks
> > > back which I think was something like:
> > >
> > > 5) Move binary dependencies - like all the icu libs - from the enhanced
> > > to the standard tree in svn and download them via http as we do other
> > > dependencies during the fetch-depends stage.
> >
> > +1. Checking out tens of megabytes of unrelated binaries is really annoying
> !
> +1
I started looking at fixing this earlier today ... making steady
progress but I've just got a new laptop so I am a little distracted now.
-Mark.
Re: [build] managing external dependencies
Posted by Alexey Petrenko <al...@gmail.com>.
2008/1/16, Alexey Varlamov <al...@gmail.com>:
> 2008/1/9, Mark Hindess <ma...@googlemail.com>:
> >
> > On 9 January 2008 at 19:07, "Alexey Varlamov" <al...@gmail.com> wrote:
> > > Folks,
> > >
> > > We did a few iterations refactoring the subject - let's have one more access
> > > ;).
> > > Why? There was a desire to have the following:
> > > 1) More fine-grained control over dependencies to empower modularity;
> > > 2) Unified approach and shared scripts across Harmony modules - VMs,
> > > classlib, jdktools;
> > > 3) Reduce duplication of resources (traffic, space, etc) between
> > > modules within the same workspace;
> > > I'd also add 4) If possible, reuse resources in different workspaces.
> > > This would be quite handy for committing purposes and for
> > > multi-platform development.
> >
> > I like all of the above but would add Tim's suggestion from a few weeks
> > back which I think was something like:
> >
> > 5) Move binary dependencies - like all the icu libs - from the enhanced
> > to the standard tree in svn and download them via http as we do other
> > dependencies during the fetch-depends stage.
>
> +1. Checking out tens of megabytes of unrelated binaries is really annoying!
+1
Re: [build] managing external dependencies
Posted by Alexey Varlamov <al...@gmail.com>.
2008/1/9, Mark Hindess <ma...@googlemail.com>:
>
> On 9 January 2008 at 19:07, "Alexey Varlamov" <al...@gmail.com> wrote:
> > Folks,
> >
> > We did a few iterations refactoring the subject - let's have one more access
> > ;).
> > Why? There was a desire to have the following:
> > 1) More fine-grained control over dependencies to empower modularity;
> > 2) Unified approach and shared scripts across Harmony modules - VMs,
> > classlib, jdktools;
> > 3) Reduce duplication of resources (traffic, space, etc) between
> > modules within the same workspace;
> > I'd also add 4) If possible, reuse resources in different workspaces.
> > This would be quite handy for committing purposes and for
> > multi-platform development.
>
> I like all of the above but would add Tim's suggestion from a few weeks
> back which I think was something like:
>
> 5) Move binary dependencies - like all the icu libs - from the enhanced
> to the standard tree in svn and download them via http as we do other
> dependencies during the fetch-depends stage.
+1. Checking out tens of megabytes of unrelated binaries is really annoying!
>
> A nice-to-have but not necessarily essential for a first approximation:
>
> 6) Ability to re-use system installed versions of dependencies - both at
> build time and ideally at run time if the dependencies are stable. For
> instance, my machine already has the 8 megs of dejavu fonts installed
> (because openoffice uses them). I fool the build in to using them to
> avoid the download but this should be made easier. At build time they
> [a subset actually] are copied into the runtime and it might be nice
> to avoid this if possible - although this part is easier to workaround
> during debian/rpm packaging.
Agree.
>
> > Geir once introduced "common_resources" module, which IMO was a step
> > in the right direction, but it was not fully adapted thus not used
> > properly. Now the new build for DRLVM is a good stimulus and
> > opportunity to accomplish the task.
> >
> > I suggest this time we deprive the classlib of it's state of Harmony
> > umbilicus and make the "common_resources" a "primordial" module ;)
>
> Does this mean the:
>
> svn co http://svn.../classlib/trunk classlib
> cd classlib
> ant fetch-depends
> ant
>
> use case will cease to work? It would be nice if we didn't have to
> break this. Perhaps we can have a classlib/trunk/common_resources
> placeholder and if a h(armon)y.common.resources properties is not defined
> then svn switch this placeholder to the correct copy (in a similar
> manner to the way the federation build works).
Well, I anticipated this point. The main issue here is locating
properties.xml: IMO we should have a sole master copy of it, unlike
current duplicates in classlib, jdktools and common_resources. It
should be possible to avoid direct import of properties.xml in
classlib/build.xml and rely on "fetch-depends" to check out
common_resources if needed. But it looks a bit ugly to me, and
probably will not preserve all current usecases?
> Users will multiple work
> spaces (or the federation build.xml) could set the hy.common.resources
> property to a shared copy?
Yes.
>
> > The idea is to move all shared properties, definitions, tasks and
> > files to the single location. Besides, the same module should be used
> > as a central repository for downloading and storing external files.
> > So, each module would "request" needed resources via standard
> > facilities (ant tasks), automatically reusing duplicates. And the same
> > "common_resources" instance can be imported to several workspaces and
> > even shared between developers.
> >
> > How?
> > 1) Roughly, the following files need to be moved from classlib to
> > common_resources:
> > make/properties.xml
> > make/depends.properties # ? move only shared items to
> > simplify maintenance
> > depends/** except depends/files/ # ? there are several libs/zips +
> > make definitions being shared
>
> As I mentioned above, I think the platform specific files should
> be moved to the standard (not enhanced tree) for download with
> fetch-depends or whatever replaces it. And we should create the
> bcprov-noidea.jar in the standard tree rather than downloading it from
> bouncycastle to comply with their request to reduce load on their
> servers.
Sure, would be nice to document this as a guideline somewhere.
And this can be done in parallel to other build refactoring, feel free
to volunteer :)
> > Closer comparison of other make/build-*.xml ant scripts in classlib
> > and jdktools may give more candidates.
> > 2) Implement tasks or macros for fetching resources, storing them and
> > providing uniform access: common_resources/download.xml svn.xml etc
> > 3) Refactor make/depends.xml in respective modules to resolve bullet #1.
> >
> > Surely the plan is very sketchy, we need to agree on general direction first.
> > Waiting for your comments/objections impatiently :)
>
> Thanks very much for kicking off this discussion.
Hope it will yield some material fruits this time...
--
Alexey
>
> -Mark.
>
>
>
>
Re: [build] managing external dependencies
Posted by Mark Hindess <ma...@googlemail.com>.
On 9 January 2008 at 19:07, "Alexey Varlamov" <al...@gmail.com> wrote:
> Folks,
>
> We did a few iterations refactoring the subject - let's have one more access
> ;).
> Why? There was a desire to have the following:
> 1) More fine-grained control over dependencies to empower modularity;
> 2) Unified approach and shared scripts across Harmony modules - VMs,
> classlib, jdktools;
> 3) Reduce duplication of resources (traffic, space, etc) between
> modules within the same workspace;
> I'd also add 4) If possible, reuse resources in different workspaces.
> This would be quite handy for committing purposes and for
> multi-platform development.
I like all of the above but would add Tim's suggestion from a few weeks
back which I think was something like:
5) Move binary dependencies - like all the icu libs - from the enhanced
to the standard tree in svn and download them via http as we do other
dependencies during the fetch-depends stage.
A nice-to-have but not necessarily essential for a first approximation:
6) Ability to re-use system installed versions of dependencies - both at
build time and ideally at run time if the dependencies are stable. For
instance, my machine already has the 8 megs of dejavu fonts installed
(because openoffice uses them). I fool the build in to using them to
avoid the download but this should be made easier. At build time they
[a subset actually] are copied into the runtime and it might be nice
to avoid this if possible - although this part is easier to workaround
during debian/rpm packaging.
> Geir once introduced "common_resources" module, which IMO was a step
> in the right direction, but it was not fully adapted thus not used
> properly. Now the new build for DRLVM is a good stimulus and
> opportunity to accomplish the task.
>
> I suggest this time we deprive the classlib of it's state of Harmony
> umbilicus and make the "common_resources" a "primordial" module ;)
Does this mean the:
svn co http://svn.../classlib/trunk classlib
cd classlib
ant fetch-depends
ant
use case will cease to work? It would be nice if we didn't have to
break this. Perhaps we can have a classlib/trunk/common_resources
placeholder and if a h(armon)y.common.resources properties is not defined
then svn switch this placeholder to the correct copy (in a similar
manner to the way the federation build works). Users will multiple work
spaces (or the federation build.xml) could set the hy.common.resources
property to a shared copy?
> The idea is to move all shared properties, definitions, tasks and
> files to the single location. Besides, the same module should be used
> as a central repository for downloading and storing external files.
> So, each module would "request" needed resources via standard
> facilities (ant tasks), automatically reusing duplicates. And the same
> "common_resources" instance can be imported to several workspaces and
> even shared between developers.
>
> How?
> 1) Roughly, the following files need to be moved from classlib to
> common_resources:
> make/properties.xml
> make/depends.properties # ? move only shared items to
> simplify maintenance
> depends/** except depends/files/ # ? there are several libs/zips +
> make definitions being shared
As I mentioned above, I think the platform specific files should
be moved to the standard (not enhanced tree) for download with
fetch-depends or whatever replaces it. And we should create the
bcprov-noidea.jar in the standard tree rather than downloading it from
bouncycastle to comply with their request to reduce load on their
servers.
> Closer comparison of other make/build-*.xml ant scripts in classlib
> and jdktools may give more candidates.
> 2) Implement tasks or macros for fetching resources, storing them and
> providing uniform access: common_resources/download.xml svn.xml etc
> 3) Refactor make/depends.xml in respective modules to resolve bullet #1.
>
> Surely the plan is very sketchy, we need to agree on general direction first.
> Waiting for your comments/objections impatiently :)
Thanks very much for kicking off this discussion.
-Mark.
Re: [build] managing external dependencies
Posted by Alexey Varlamov <al...@gmail.com>.
2008/1/9, Gregory Shimansky <gs...@apache.org>:
> Alexey Varlamov said the following on 09.01.2008 16:07:
> > Folks,
> >
> > We did a few iterations refactoring the subject - let's have one more access ;).
> > Why? There was a desire to have the following:
> > 1) More fine-grained control over dependencies to empower modularity;
> > 2) Unified approach and shared scripts across Harmony modules - VMs,
> > classlib, jdktools;
> > 3) Reduce duplication of resources (traffic, space, etc) between
> > modules within the same workspace;
> > I'd also add 4) If possible, reuse resources in different workspaces.
> > This would be quite handy for committing purposes and for
> > multi-platform development.
> >
> > Geir once introduced "common_resources" module, which IMO was a step
> > in the right direction, but it was not fully adapted thus not used
> > properly. Now the new build for DRLVM is a good stimulus and
> > opportunity to accomplish the task.
> >
> > I suggest this time we deprive the classlib of it's state of Harmony
> > umbilicus and make the "common_resources" a "primordial" module ;)
> > The idea is to move all shared properties, definitions, tasks and
> > files to the single location. Besides, the same module should be used
> > as a central repository for downloading and storing external files.
> > So, each module would "request" needed resources via standard
> > facilities (ant tasks), automatically reusing duplicates. And the same
> > "common_resources" instance can be imported to several workspaces and
> > even shared between developers.
>
> Yes I think it would be very nice to have, especially for creating
> source packages integrated into Linux distributions. Then files may be
> reused from the already installed on the system.
>
> One thing I wonder about, if an external dependency is downloaded in
> source form, then common_resources would have to build it as well. E.g.
> drlvm build needs APR (actually it also patches it) and LOG4CXX. Do you
> think common_resources should contain only sources for APR or the
> already compiled binary?
Sorry for the late reply. Indeed it would be reasonable to provide
ready binaries whenever possible. This can be good enhancement in
future, and for the first approximation getting sources only should be
enough. Moreover, currently DRLVM uses non-standard build of APR which
complicates things.
>
> > How?
> > 1) Roughly, the following files need to be moved from classlib to
> > common_resources:
> > make/properties.xml
> > make/depends.properties # ? move only shared items to
> > simplify maintenance
> > depends/** except depends/files/ # ? there are several libs/zips +
> > make definitions being shared
> > Closer comparison of other make/build-*.xml ant scripts in classlib
> > and jdktools may give more candidates.
> > 2) Implement tasks or macros for fetching resources, storing them and
> > providing uniform access: common_resources/download.xml svn.xml etc
> > 3) Refactor make/depends.xml in respective modules to resolve bullet #1.
> >
> > Surely the plan is very sketchy, we need to agree on general direction first.
> > Waiting for your comments/objections impatiently :)
>
> I think that the change is good. Thank you very much for your effort in
> improving current build situation!
Thank you too for the kind words!
--
Alexey
>
> --
> Gregory
>
>
Re: [build] managing external dependencies
Posted by Gregory Shimansky <gs...@apache.org>.
Alexey Varlamov said the following on 09.01.2008 16:07:
> Folks,
>
> We did a few iterations refactoring the subject - let's have one more access ;).
> Why? There was a desire to have the following:
> 1) More fine-grained control over dependencies to empower modularity;
> 2) Unified approach and shared scripts across Harmony modules - VMs,
> classlib, jdktools;
> 3) Reduce duplication of resources (traffic, space, etc) between
> modules within the same workspace;
> I'd also add 4) If possible, reuse resources in different workspaces.
> This would be quite handy for committing purposes and for
> multi-platform development.
>
> Geir once introduced "common_resources" module, which IMO was a step
> in the right direction, but it was not fully adapted thus not used
> properly. Now the new build for DRLVM is a good stimulus and
> opportunity to accomplish the task.
>
> I suggest this time we deprive the classlib of it's state of Harmony
> umbilicus and make the "common_resources" a "primordial" module ;)
> The idea is to move all shared properties, definitions, tasks and
> files to the single location. Besides, the same module should be used
> as a central repository for downloading and storing external files.
> So, each module would "request" needed resources via standard
> facilities (ant tasks), automatically reusing duplicates. And the same
> "common_resources" instance can be imported to several workspaces and
> even shared between developers.
Yes I think it would be very nice to have, especially for creating
source packages integrated into Linux distributions. Then files may be
reused from the already installed on the system.
One thing I wonder about, if an external dependency is downloaded in
source form, then common_resources would have to build it as well. E.g.
drlvm build needs APR (actually it also patches it) and LOG4CXX. Do you
think common_resources should contain only sources for APR or the
already compiled binary?
> How?
> 1) Roughly, the following files need to be moved from classlib to
> common_resources:
> make/properties.xml
> make/depends.properties # ? move only shared items to
> simplify maintenance
> depends/** except depends/files/ # ? there are several libs/zips +
> make definitions being shared
> Closer comparison of other make/build-*.xml ant scripts in classlib
> and jdktools may give more candidates.
> 2) Implement tasks or macros for fetching resources, storing them and
> providing uniform access: common_resources/download.xml svn.xml etc
> 3) Refactor make/depends.xml in respective modules to resolve bullet #1.
>
> Surely the plan is very sketchy, we need to agree on general direction first.
> Waiting for your comments/objections impatiently :)
I think that the change is good. Thank you very much for your effort in
improving current build situation!
--
Gregory