You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Mark Hindess <ma...@googlemail.com> on 2006/05/09 21:46:15 UTC
Re: Supporting working on a single module?
On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>
> Mark Hindess wrote:
> >
> > As the Harmony Classlib grows, I think that being able to work on a
> > single module (or some subset of the modules) will become the
> > typical way of working for many (perhaps even most) contributors.
>
> Sure, that makes sense.
>
> > So I think we need a plan to support this. I also think that
> > forcing ourselves to support this mode of working sooner rather than
> > later will help us keep from accidentally breaking the modularity in
> > the current build/development process.
>
> Can you not work on a single module now?
Yes. But only by checking out everything.
> > Oliver is currently looking at restructuring the native source to
> > the appropriate modules/*/src/main/native directory.
>
> Can he talk about it out here,
He/We did - more than once. For instance:
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
There are quite a lot of issues when you think about how to get from
where we are today to where we want to be. It's not as trivial as it
sounds so even writing a good description of the issue(s) for the list
is not straightforward.
> or is this for your internal use?
Definitely not. We sometimes talk to each other off-list but not very
often! ;-)
> > One question that comes out of this investigation is: where should
> > the include files "live"? I think they belong with the module that
> > provides the implementation defined by the declarations in the
> > header. That is, zipsup.h is "owned" by archive, hythread.h is
> > "owned" by thread (luni), etc.
> Sure - that makes sense.
> > However, if the build was to refer to the header files within the
> > modules then this breaks the modularity. So, for example, someone
> > working on a single module would have to checkout all the dependent
> > modules rather than just the one they wish to work on.
> >
> > So, perhaps, at build time we should copy the header files out of
> > the owning module into a common include directory. All modules
> > would then refer to the header file in the common include directory.
You didn't comment on this bit.
I don't think it's a good idea for one module referencing into another
directly where it isn't a well-defined interface that we can
manage/enforce appropriately. Would you agree?
We might as well try to agree how to construct and manage this
"interface". I think the copying/"hdk" idea is a good solution.
I didn't imagine these issues would be this contentious or perhaps I'd
have tried to separate these two (related) issues.
> > This means we can then create a snapshot tar/zip of the common
> > include directory which someone wishing to work on a single module
> > could download to build against. This is not dissimilar to the
> > current way in which you could build the java sources against a
> > deploy/jre snapshot.
> Why wouldn't someone wishing to work on a single module just checkout
> all the code?
So someone working on prefs (which is approximately 2MB) would need to
check out all the source for luni (the current largest module at ~36MB),
awt, swing, sound, etc. ?
I usually have half a dozen workspaces where I'm trying things out (even
more at the moment since I'm looking at the four contributions that are
being voted on). This isn't too bad at the moment with each one being
about 1/4 GB but it will get bigger over time and therefore less
manageable.
> I'm really wary about having non-SVN-sourced artifacts meant to be
> used in building/development.
Isn't that modularity? Why shouldn't we do it - our customers will be?
> Smells a bit like the first step towards the classic Windows
> "dll-hell".
Undoubtedly, with people working on separate modules, we will get build
breaks. But we'll get them when, for example, we don't have sufficient
unit test coverage within the module being worked on - we had an example
of this not long ago IIRC. We'll also have breaks when people have made
different assumptions about the meaning of the spec or the definition of
the internal API. It is a good thing that we find these *bugs* within
the project... being our own customers!
I think this will actually help *avoid* some of the problems you are
thinking about.
> > For windows, the snapshot would also include the .lib files that are
> > required to build for the run-time .dll files.
> >
> > What is this new snapshot? Well, Tim suggested
>
> (Where?)
>
> > that it was a little like the jdk but for Harmony development. So
> > perhaps it should be called an hdk, Harmony Development Kit?
>
> I'm missing the point... why wouldn't a developer checkout head, build
> that, and then get to work on whatever part of whatever module they
> wanted?
The classlib is getting pretty big. If we are serious about modularity,
then we should try to support it right through from build, development,
deployment and ultimately at runtime.
> > Logically, in the same way that a jdk contains a jre, the hdk should
> > contain the jdk (which will contain the jre). Thus, we'd have a
> > hierarchy like:
> >
> > deploy/hdk
> > deploy/hdk/jdk
> > deploy/hdk/jdk/jre
> >
> > When we come to create snapshots/releases, it's easy to see how we'd
> > create archives for the hdk, jdk, and jre.
> >
> > Unfortunately, though I think this solution is quite elegant, there
> > are quite a few references to the "deploy/jre" path that would need to
> > be fixed. However, I think this is something we should discuss and, if
> > we decide it is the right thing to do, then we should take the hit and
> > move things around now.
> >
> > What do you think?
>
> I don't understand the basic problem you're trying to solve... Maybe
> because this is the first I've heard of any problems related to this,
> or I wasn't paying attention.
Oliver, Tim and I had a very brief conversation about this "over coffee"
and I said I'd raise it on the list. Obviously I didn't do a very good
job of it. Sorry.
> Is it that since modules reference other modules, working in one
> module means you need the dependent modules around?
Yes, exactly. We shouldn't require users to have the entire "source"
for all the dependent modules around. We should support modularity at
the development level. (Like we do already with the stubs for the
luni/security kernel classes. We don't require developers to have the
VM around.)
When I'm developing C code I reference the libc header files but I don't
go poking around including random headers from the libc source.
So if I'm working on sql, I don't see why I shouldn't develop using
header files and other well-defined parts of the API for other modules
rather than having to have all of the source code checked out.
Of course, we should still support people working by checking out
everything but we shouldn't require it.
> A long time ago we talked about something like this, a pre-build step
> that produces a "hdk"-like set of artifacts that a developer could
> then use if they were focused down inside a module.
>
> Is this the same thing returning for discussion?
Not since I've been actively following the list but I'll dig about in
the archives later.
> Couldn't we just use a standard snapshot? Take the latest nightly,
> drop into some magical place in the tree, and then go into your module
> and work from there?
Well, I was suggesting snapshots that might be:
1) hdk (inc. jdk and jre)
2) jdk (inc. jre)
3) jre
but I think (?) you are suggesting using the one I overlooked:
0) everything in classlib/trunk
I think 0) is going to get pretty big (but we should still create it)
and I think we should actively support using 1) for development too.
Wasn't someone recently pleading with Eclipse to make smaller artifacts?
;-)
Regards,
Mark.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Chris Gray wrote:
> On Tuesday 09 May 2006 23:26, Tim Ellison wrote:
>> Geir Magnusson Jr wrote:
>
>>> I guess the question for me, now that I thought about it longer, is
>>> where would we use the separate stub library? I can see us filling in
>>> our blanks w/ stubs for whatever reason (totally fooling JAPI :) and
>>> getting people to flesh those out...
>> Plenty of people like to work that way (though it is not my personal
>> preference).
>
> Mine neither, precisely because of the JAPI-fooling aspect. That's why I
> actually rather like the idea of a stand-alone 100% stubbed version, distinct
> from the real (but possibly incomplete) implementation.
I guess it wouldn't hurt anything if kept outside of the classlib/ tree.
I wonder how well -formed the javadoc is...
geir
>
> Chris
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Chris Gray <ch...@kiffer.be>.
On Tuesday 09 May 2006 23:26, Tim Ellison wrote:
> Geir Magnusson Jr wrote:
> > I guess the question for me, now that I thought about it longer, is
> > where would we use the separate stub library? I can see us filling in
> > our blanks w/ stubs for whatever reason (totally fooling JAPI :) and
> > getting people to flesh those out...
>
> Plenty of people like to work that way (though it is not my personal
> preference).
Mine neither, precisely because of the JAPI-fooling aspect. That's why I
actually rather like the idea of a stand-alone 100% stubbed version, distinct
from the real (but possibly incomplete) implementation.
Chris
--
Chris Gray /k/ Embedded Java Solutions BE0503765045
Embedded & Mobile Java, OSGi http://www.k-embedded-java.com/
chris.gray@kiffer.be +32 3 216 0369
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr wrote:
> Tim Ellison wrote:
>> I disagree -- we spent a good period of time last summer carving up the
>> class library into modules defined by Java and internal APIs. I believe
>> it would be detrimental to disregard these boundaries by compiling
>> against the entire Java APIs, as that would perpetuate the 'spaghetti
>> code syndrome'.
>
> Aside : there's no harm in a "rt.jar" created from our modules. Are
> modules are there for development convenience? it's plausible that
> we'll have a monolithic classlib distro as well as the modular one, right?
For sure, we have full freedom on how it is ultimately packaged.
>> We could use compile-against stubs, but would also need them for the
>> org.apache.* packages that comprise our internal APIs, for now I see no
>> problem with using the actual JARs that we produce.
>>
>
> I guess the question for me, now that I thought about it longer, is
> where would we use the separate stub library? I can see us filling in
> our blanks w/ stubs for whatever reason (totally fooling JAPI :) and
> getting people to flesh those out...
Plenty of people like to work that way (though it is not my personal
preference).
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Tim Ellison wrote:
> Chris Gray wrote:
>> On Tuesday 09 May 2006 22:03, Etienne Gagnon wrote:
>>> Yes. But only by checking out everything.
>>>
>>> I am developing a API "stubs" project, which is a full "stubs"
>>> implementation of the Java 1.5 API. My objective was actually to allow
>>> for not needing an API "implementation" to compile code against the API.
>>> I was planning to use this, among other uses, for compiling SableVM's
>>> luni-kernel implementation.
>> For what my opinion is worth (on a good day, a cup of coffee, but not at Caffè
>> Florian), this would be an excellent thing to have. It will never be easy to
>> work on the core Java APIs in a totally modular way (because Sun didn't
>> design things that way), but with such a set of stubs one could at least work
>> on a group of classes in isolation and be able to compile them to bytecode.
>> Furthermore the stubs can readily be used for white-box testing during
>> development, by simply adding println()s. Go for it!
>
> I disagree -- we spent a good period of time last summer carving up the
> class library into modules defined by Java and internal APIs. I believe
> it would be detrimental to disregard these boundaries by compiling
> against the entire Java APIs, as that would perpetuate the 'spaghetti
> code syndrome'.
Aside : there's no harm in a "rt.jar" created from our modules. Are
modules are there for development convenience? it's plausible that
we'll have a monolithic classlib distro as well as the modular one, right?
>
> We could use compile-against stubs, but would also need them for the
> org.apache.* packages that comprise our internal APIs, for now I see no
> problem with using the actual JARs that we produce.
>
I guess the question for me, now that I thought about it longer, is
where would we use the separate stub library? I can see us filling in
our blanks w/ stubs for whatever reason (totally fooling JAPI :) and
getting people to flesh those out...
geir
> Regards,
> Tim
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Tim Ellison <t....@gmail.com>.
Chris Gray wrote:
> On Tuesday 09 May 2006 22:03, Etienne Gagnon wrote:
>> Yes. But only by checking out everything.
>>
>> I am developing a API "stubs" project, which is a full "stubs"
>> implementation of the Java 1.5 API. My objective was actually to allow
>> for not needing an API "implementation" to compile code against the API.
>> I was planning to use this, among other uses, for compiling SableVM's
>> luni-kernel implementation.
>
> For what my opinion is worth (on a good day, a cup of coffee, but not at Caffè
> Florian), this would be an excellent thing to have. It will never be easy to
> work on the core Java APIs in a totally modular way (because Sun didn't
> design things that way), but with such a set of stubs one could at least work
> on a group of classes in isolation and be able to compile them to bytecode.
> Furthermore the stubs can readily be used for white-box testing during
> development, by simply adding println()s. Go for it!
I disagree -- we spent a good period of time last summer carving up the
class library into modules defined by Java and internal APIs. I believe
it would be detrimental to disregard these boundaries by compiling
against the entire Java APIs, as that would perpetuate the 'spaghetti
code syndrome'.
We could use compile-against stubs, but would also need them for the
org.apache.* packages that comprise our internal APIs, for now I see no
problem with using the actual JARs that we produce.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Chris Gray <ch...@kiffer.be>.
Geir scripsit:
> > For what my opinion is worth (on a good day, a cup of coffee, but not at
> > Caffè Florian), this would be an excellent thing to have. It will never
> > be easy to work on the core Java APIs in a totally modular way (because
> > Sun didn't design things that way), but with such a set of stubs one
> > could at least work on a group of classes in isolation and be able to
> > compile them to bytecode. Furthermore the stubs can readily be used for
> > white-box testing during development, by simply adding println()s. Go for
> > it!
>
> 1) I wonder if we could just add those println's w/ AOP since we're
> fundamentally lazy.
I imagined that these println's would be ad-hoc and ephemeral, a step along
the way to producing other deliverables (the class libraries themselves, and
the various kinds of test that have already been discussed here). If you're
developing method M of class C and your implementation invokes method M' of
class C', you sometimes want to have a M' which just prints out "hello, I've
been called with inputs foo, bar and baz" and/or returns some prearranged
result; but I don't think you'd want the stub method to have this feature
built-in for every conceivable invocation. Not unless you embark on an
ambitious reflection-based framework with scripting capabilities, which of
course could be huge fun in itself. :)
> 2) Could we mechanically create the stubs, rather than having a parallel
> source tree? One might argue that you could generate such a stub by
> transforming the spec javadoc to code.... no muss, no fuss, and darn
> quick...
This is not such a daft idea. If it's not literally possible (and Etienne can
mostly likely tell us why not), then it should still be possible to perform
extensive automatic automatic checking of the stubs using japitools or
similar.
Chris
--
Chris Gray /k/ Embedded Java Solutions BE0503765045
Embedded & Mobile Java, OSGi http://www.k-embedded-java.com/
chris.gray@kiffer.be +32 3 216 0369
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Chris Gray wrote:
> On Tuesday 09 May 2006 22:03, Etienne Gagnon wrote:
>> Yes. But only by checking out everything.
>>
>> I am developing a API "stubs" project, which is a full "stubs"
>> implementation of the Java 1.5 API. My objective was actually to allow
>> for not needing an API "implementation" to compile code against the API.
>> I was planning to use this, among other uses, for compiling SableVM's
>> luni-kernel implementation.
>
> For what my opinion is worth (on a good day, a cup of coffee, but not at Caffè
> Florian), this would be an excellent thing to have. It will never be easy to
> work on the core Java APIs in a totally modular way (because Sun didn't
> design things that way), but with such a set of stubs one could at least work
> on a group of classes in isolation and be able to compile them to bytecode.
> Furthermore the stubs can readily be used for white-box testing during
> development, by simply adding println()s. Go for it!
1) I wonder if we could just add those println's w/ AOP since we're
fundamentally lazy.
2) Could we mechanically create the stubs, rather than having a parallel
source tree? One might argue that you could generate such a stub by
transforming the spec javadoc to code.... no muss, no fuss, and darn
quick...
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Chris Gray <ch...@kiffer.be>.
On Tuesday 09 May 2006 22:03, Etienne Gagnon wrote:
> Yes. But only by checking out everything.
>
> I am developing a API "stubs" project, which is a full "stubs"
> implementation of the Java 1.5 API. My objective was actually to allow
> for not needing an API "implementation" to compile code against the API.
> I was planning to use this, among other uses, for compiling SableVM's
> luni-kernel implementation.
For what my opinion is worth (on a good day, a cup of coffee, but not at Caffè
Florian), this would be an excellent thing to have. It will never be easy to
work on the core Java APIs in a totally modular way (because Sun didn't
design things that way), but with such a set of stubs one could at least work
on a group of classes in isolation and be able to compile them to bytecode.
Furthermore the stubs can readily be used for white-box testing during
development, by simply adding println()s. Go for it!
Chris
--
Chris Gray /k/ Embedded Java Solutions BE0503765045
Embedded & Mobile Java, OSGi http://www.k-embedded-java.com/
chris.gray@kiffer.be +32 3 216 0369
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Etienne Gagnon <eg...@sablevm.org>.
Hi Mark,
Mark Hindess wrote:
>>Can you not work on a single module now?
>
> Yes. But only by checking out everything.
I am developing a API "stubs" project, which is a full "stubs"
implementation of the Java 1.5 API. My objective was actually to allow
for not needing an API "implementation" to compile code against the API.
I was planning to use this, among other uses, for compiling SableVM's
luni-kernel implementation.
Maybe I could contribute the stubs to Harmony, and you could use them to
achieve your goal?
Note: By stubs, I mean, Java sources that, when compiled, produce .class
files which export the specified API. Of course, they don't implement
the functionality. You can have a look at the preliminary work:
svn co svn://svn.sablevm.org/classlib-stubs/trunk/
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: Supporting working on a single module?
Posted by "Jimmy, Jing Lv" <fi...@gmail.com>.
> Oliver Deakin wrote:
> Geir Magnusson Jr wrote:
>> Mark Hindess wrote:
>>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>>> Mark Hindess wrote:
>>>>> As the Harmony Classlib grows, I think that being able to work on a
>>>>> single module (or some subset of the modules) will become the
>>>>> typical way of working for many (perhaps even most) contributors.
>>>> Sure, that makes sense.
>>>>
>>>>> So I think we need a plan to support this. I also think that
>>>>> forcing ourselves to support this mode of working sooner rather than
>>>>> later will help us keep from accidentally breaking the modularity in
>>>>> the current build/development process.
>>>> Can you not work on a single module now?
>>>
>>> Yes. But only by checking out everything.
>>
>> Wouldn't you do that anyway?
>>
>> I guess thats the mystery to me. I figured that someone would do the
>> full checkout, and go from there.
>
> I think it's good if we can cater not only for those developers who want
> to completely checkout classlib trunk, but also for those who want to
> work only on a single module. When I say "work on a single module", I
> mean only checking that module's directory out of SVN, and not the code
> for all the other modules, thereby saving download time and disk space,
> not to mention the subsequent reduced build times of only recompiling a
> single module.
>
> Currently a developer cannot work on both the native and Java code for a
> module without checking out the whole of classlib trunk, because the
> native source directories and Ant build script associated with them are
> completely separate from the modules. IMHO it would be great if we could
> get the build scripts and code repository layout into a state where a
> developer can just checkout a module, grab an hdk from some location and
> start changing and rebuilding both Java and native code (and also
> running tests against the rebuilt libraries).
>
First of all, I shall give warm welcome to the refactor of native code.
In fact when I want to touch native code, I download "native-source" and
"depends" form svn, force a full build with ant using build.xml in
native-source/. After that, when I modify some code in a certain part,
e.g. luni, I change directory to luni and build that single module with
"make" (or "nmake" in windows), in this way I get a new hyluni .DLL or
.so file, copy it to harmony/deploy/deploy/jre/bin/ and restart VM.
Separate them into modules would surely make it easy for developers.
> To do this there are at least three steps needed, as far as I can see:
>
> 1) Refactor the native code into the modular structure we currently have
> for Java code and tests. This has been spoken about before at [1]. The
> native code would be located within each module at modules/<module
> name>/src/main/native. As a starting point, can I propose that the
> natives be broken down in the following way:
>
> modules/archive/src/main/native/
> |-------archive/
> |-------zip/
>
> +------zlib/
> modules/auth/src/main/native/
> +-------auth/
>
> modules/luni/src/main/native/
> |--------common/
> |--------fdlibm/
> |--------launcher/
> |--------luni/
> |--------pool/
> |--------port/
> |--------sig/
> |--------thread/
> |--------vmi/
> +-------vmls/
>
> modules/prefs/src/main/native/
> +-------prefs/
>
> modules/text/src/main/native/
> |-------text/
> +------unicode/ (comes from the
> icu4c zips stored currently in depends)
>
> Additionally, each module may have an include directory that contains
> header files required across its native components. Each native
> component would contain subdirectories for its shared and platform
> specific code.
>
If I do not misunderstand the chart I'm afraid combining native and java
code into one module would not be a good idea, specially for module
Luni. It IS very large now, and become much larger if we do so, thus it
may spend long time to download from SVN.
What's more important, many user may care nothing about the native code,
they only want to read or modify the java code. Add native code to the
module would be an encumbrance to them. So, why not put them alone in a
native directory as (it seems nearly the same with current)
+------zlib/
modules/native/auth/src/
+-------auth/
modules/native/luni/src/
|--------common/
|--------fdlibm/
...
and alter build scripts? Perhaps only scripts are important to rewrite.
If I am right, what we really need is: we can download one single module
native source, e.g., Luni, and all other stubs, libs, DLL/so files or
something, then modify Luni native source, and build them all and create
a new hyluni.dll/so. In this way we no longer need to download other
source code besides Luni but still get a success build.
> 2) Alter the global build scripts to start creating an hdk under the
> deploy directory, which contains a jdk which contains a jre. As Mark
> described before, this hdk would probably look something like:
>
> deploy/hdk/
> |--------jdk/
> | |-------jre/ (already being created at
> the moment as deploy/jre)
> | +------include/ (already being created
> at the moment as deploy/include)
> |--------include/
> +-------lib/
>
> The hdk/include directory would be much like the current
> native-src/<platform>/include directories, and would contain a minimal
> set of header files that define an interface between native modules. It
> would be the responsibility of the individual module to copy its
> required headers to this directory as a prerequisite for building.
> The hdk/lib directory would exist on Windows platforms, and would
> contain .lib files for each native component.
>
> An hdk snapshot would then be a zip of the hdk directory, and should
> provide everything necessary for a developer to build both native and
> Java code against.
>
> 3) Modularise native and Java build scripts so that the individual
> module's Ant scripts are capable of building native and Java source
> against an hdk, rather than having its code compiled as part of a global
> build. The rebuilt libraries would be placed back into the hdk
> directory, so that the developers local hdk always contains their latest
> built code (much as the current deploy/jre does).
>
>
> Additionally, I agree with Nathan [2] that allowing a developer working
> in a modular way to run the entire test suite against their
> work-in-progress code would be a good thing, and this could be done by
> bundling tests and their resources in with an hdk.
>
> Regards,
> Oliver
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
>
> [2]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
>
>
--
Best Regards!
Jimmy, Jing Lv
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Tim Ellison wrote:
> That layout works for me too.
>
> Patches welcome ;-)
>
hehe, had a feeling that might be coming ;)
Im actually working on a couple of preliminary patches at the moment for
this refactoring. One is a minor tidy up, raised in HARMONY-451. I hope
to raise another JIRA soon which will alter the build scripts to create
and populate an hdk directory structure under deploy - which basically
means moving the deploy/jre and deploy/include directories down a level
to deploy/jdk/jre and deploy/jdk/include. Once this is done we will be
in a good position to start moving the native code into modules (which
Im also happy to work on and provide patches for).
> Regards,
> Tim
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Tim Ellison <t....@gmail.com>.
That layout works for me too.
Patches welcome ;-)
Regards,
Tim
Oliver Deakin wrote:
> Andrey Chernyshev wrote:
>>> I was thinking that platform specific directories would be laid out
>>> underneath each native
>>> component directory. So, for example, underneath the
>>> modules/luni/src/main/native/port
>>> directory there would be the following structure (avoiding ascii tree
>>> diagrams):
>>>
>>> modules/luni/src/main/native/port/shared
>>> modules/luni/src/main/native/port/linux
>>> modules/luni/src/main/native/port/windows
>>>
>>> with further platform specific directories being added as we expand.
>>
>> Yes, I was thinking about that too, but didn't mention :).
>> I remember there was a discussion about this sometime in the past [1],
>> it looked like most people agreed at that time that keeping OSes &
>> platforms as the directory names is the preferred choice.
>>
>
> Yes, I think you're right. At the moment the layout is quite simple
> since we only have
> two platforms.
>
> When we start to expand our platform list, I believe the layout that you
> linked in [1] is
> suitably descriptive and simple to use, with directory names
> incorporating OS as the
> first level of code specialization and architecture the second,
> separated by an underscore.
> I envisage that eventually we might have a layout similar to (hope this
> diagram
> works - all subdirs under <component> are at the same level):
>
> modules/<module>/src/main/native/<component>/
>
> |----------shared/
>
> |----------aix/
>
> |----------linux/
>
> |----------linux_amd/
>
> |----------linux_ppc/
>
> |----------linux_s390/
>
> |----------linux_x86/
>
> |----------solaris/
>
> |----------solaris_x86/
>
> |----------windows/
>
> |----------windows_amd/
>
> |----------windows_x86/
>
> |----------unix/
>
> |----------zos/
>
> |----------shared_include/
>
> |----------windows_include/
>
> \----------unix_include/
>
>
> Regards,
> Oliver
>
>
>> Thanks,
>> Andrey.
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
>>
>>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Andrey Chernyshev wrote:
>> I was thinking that platform specific directories would be laid out
>> underneath each native
>> component directory. So, for example, underneath the
>> modules/luni/src/main/native/port
>> directory there would be the following structure (avoiding ascii tree
>> diagrams):
>>
>> modules/luni/src/main/native/port/shared
>> modules/luni/src/main/native/port/linux
>> modules/luni/src/main/native/port/windows
>>
>> with further platform specific directories being added as we expand.
>
> Yes, I was thinking about that too, but didn't mention :).
> I remember there was a discussion about this sometime in the past [1],
> it looked like most people agreed at that time that keeping OSes &
> platforms as the directory names is the preferred choice.
>
Yes, I think you're right. At the moment the layout is quite simple
since we only have
two platforms.
When we start to expand our platform list, I believe the layout that you
linked in [1] is
suitably descriptive and simple to use, with directory names
incorporating OS as the
first level of code specialization and architecture the second,
separated by an underscore.
I envisage that eventually we might have a layout similar to (hope this
diagram
works - all subdirs under <component> are at the same level):
modules/<module>/src/main/native/<component>/
|----------shared/
|----------aix/
|----------linux/
|----------linux_amd/
|----------linux_ppc/
|----------linux_s390/
|----------linux_x86/
|----------solaris/
|----------solaris_x86/
|----------windows/
|----------windows_amd/
|----------windows_x86/
|----------unix/
|----------zos/
|----------shared_include/
|----------windows_include/
\----------unix_include/
Regards,
Oliver
> Thanks,
> Andrey.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Andrey Chernyshev <a....@gmail.com>.
> I was thinking that platform specific directories would be laid out
> underneath each native
> component directory. So, for example, underneath the
> modules/luni/src/main/native/port
> directory there would be the following structure (avoiding ascii tree
> diagrams):
>
> modules/luni/src/main/native/port/shared
> modules/luni/src/main/native/port/linux
> modules/luni/src/main/native/port/windows
>
> with further platform specific directories being added as we expand.
Yes, I was thinking about that too, but didn't mention :).
I remember there was a discussion about this sometime in the past [1],
it looked like most people agreed at that time that keeping OSes &
platforms as the directory names is the preferred choice.
Thanks,
Andrey.
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
On 5/15/06, Oliver Deakin <ol...@googlemail.com> wrote:
>
>
> Tim Ellison wrote:
> > Andrey Chernyshev wrote:
> > <snip>
> >
> >
> >> (1) Do we need to keep the 'main' directory in every module? If we
> >> need to have a distinction between tests and sources, may be just pull
> >> tests one level up and have something like:
> >> archive/
> >> src/
> >> native/
> >> java/
> >> tests/
> >> native/
> >> java/
> >> I wonder if 'main' is an extra level of the directory tree we can
> >> actually avoid of (lazy people don't like typing cd too much :))
> >>
> >
> > Really lazy people use path completion and don't care ;-)
> >
> >
> >> (2) Why do we need to have 'luni' two times in the path, e.g.
> >> modules/luni/src/main/native/luni/ ? If we need to put an additional
> >> stuff like 'port' to the luni module, perhaps it could be just enough
> >> to put it into a subdirectory within native, e.g:
> >> modules/luni/src/native/port/ ?
> >>
> >
> > Is it just the name of that path element that you object to? Seems a
> > bit cleaner to me if there is a bucket to put that source in.
> >
> > However, (comment to Oliver as well), I'm left wondering where the
> > platform-specific vs. common code distinction is made?
> >
>
> I was thinking that platform specific directories would be laid out
> underneath each native
> component directory. So, for example, underneath the
> modules/luni/src/main/native/port
> directory there would be the following structure (avoiding ascii tree
> diagrams):
>
> modules/luni/src/main/native/port/shared
> modules/luni/src/main/native/port/linux
> modules/luni/src/main/native/port/windows
>
> with further platform specific directories being added as we expand.
>
>
> >
> >> BTW, I've noticed that this proposal is very similar to the DRLVM
> >> source tree organization, which is like:
> >>
> >
> > Great minds and all that :-)
> >
> >
> >> - vm
> >> - include - top level include which contains h files exported by
> >> various VM components;
> >> - interpreter
> >> - jitrino
> >> - vmcore
> >> ...
> >> <other VM components>
> >>
> >> The module vmcore, for example, contains both native and java code:
> >> vmcore/src/kernel_classes
> >> - native
> >> - javasrc
> >>
> >> The building system for DRLVM has been designed in a modular way as well:
> >> There is a "building engine part at the build/make and
> >> build/make/targets directory which is shared by all components,
> >> Each VM module has a building descriptor which is currently located at
> >> build/make/components directory, but can also be easily moved to the
> >> component source tree to support the idea of full independent checkout
> >> of a specific module.
> >>
> >> I think the building system provided with DRLVM can easily be used to
> >> support the source modularization approach, the proposed 'hdk' in that
> >> case would provide the developers, besides the "public includes", with
> >> the shared part of the building scripts as well.
> >>
> >
> > We should continue to collaborate on finding the best solution -- it has
> > worked very well so far!
> >
>
> Agreed!
>
> > Regards,
> > Tim
> >
> >
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Tim Ellison wrote:
> Andrey Chernyshev wrote:
> <snip>
>
>
>> (1) Do we need to keep the 'main' directory in every module? If we
>> need to have a distinction between tests and sources, may be just pull
>> tests one level up and have something like:
>> archive/
>> src/
>> native/
>> java/
>> tests/
>> native/
>> java/
>> I wonder if 'main' is an extra level of the directory tree we can
>> actually avoid of (lazy people don't like typing cd too much :))
>>
>
> Really lazy people use path completion and don't care ;-)
>
>
>> (2) Why do we need to have 'luni' two times in the path, e.g.
>> modules/luni/src/main/native/luni/ ? If we need to put an additional
>> stuff like 'port' to the luni module, perhaps it could be just enough
>> to put it into a subdirectory within native, e.g:
>> modules/luni/src/native/port/ ?
>>
>
> Is it just the name of that path element that you object to? Seems a
> bit cleaner to me if there is a bucket to put that source in.
>
> However, (comment to Oliver as well), I'm left wondering where the
> platform-specific vs. common code distinction is made?
>
I was thinking that platform specific directories would be laid out
underneath each native
component directory. So, for example, underneath the
modules/luni/src/main/native/port
directory there would be the following structure (avoiding ascii tree
diagrams):
modules/luni/src/main/native/port/shared
modules/luni/src/main/native/port/linux
modules/luni/src/main/native/port/windows
with further platform specific directories being added as we expand.
>
>> BTW, I've noticed that this proposal is very similar to the DRLVM
>> source tree organization, which is like:
>>
>
> Great minds and all that :-)
>
>
>> - vm
>> - include - top level include which contains h files exported by
>> various VM components;
>> - interpreter
>> - jitrino
>> - vmcore
>> ...
>> <other VM components>
>>
>> The module vmcore, for example, contains both native and java code:
>> vmcore/src/kernel_classes
>> - native
>> - javasrc
>>
>> The building system for DRLVM has been designed in a modular way as well:
>> There is a "building engine part at the build/make and
>> build/make/targets directory which is shared by all components,
>> Each VM module has a building descriptor which is currently located at
>> build/make/components directory, but can also be easily moved to the
>> component source tree to support the idea of full independent checkout
>> of a specific module.
>>
>> I think the building system provided with DRLVM can easily be used to
>> support the source modularization approach, the proposed 'hdk' in that
>> case would provide the developers, besides the "public includes", with
>> the shared part of the building scripts as well.
>>
>
> We should continue to collaborate on finding the best solution -- it has
> worked very well so far!
>
Agreed!
> Regards,
> Tim
>
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Tim Ellison <t....@gmail.com>.
Andrey Chernyshev wrote:
<snip>
> (1) Do we need to keep the 'main' directory in every module? If we
> need to have a distinction between tests and sources, may be just pull
> tests one level up and have something like:
> archive/
> src/
> native/
> java/
> tests/
> native/
> java/
> I wonder if 'main' is an extra level of the directory tree we can
> actually avoid of (lazy people don't like typing cd too much :))
Really lazy people use path completion and don't care ;-)
> (2) Why do we need to have 'luni' two times in the path, e.g.
> modules/luni/src/main/native/luni/ ? If we need to put an additional
> stuff like 'port' to the luni module, perhaps it could be just enough
> to put it into a subdirectory within native, e.g:
> modules/luni/src/native/port/ ?
Is it just the name of that path element that you object to? Seems a
bit cleaner to me if there is a bucket to put that source in.
However, (comment to Oliver as well), I'm left wondering where the
platform-specific vs. common code distinction is made?
> BTW, I've noticed that this proposal is very similar to the DRLVM
> source tree organization, which is like:
Great minds and all that :-)
> - vm
> - include - top level include which contains h files exported by
> various VM components;
> - interpreter
> - jitrino
> - vmcore
> ...
> <other VM components>
>
> The module vmcore, for example, contains both native and java code:
> vmcore/src/kernel_classes
> - native
> - javasrc
>
> The building system for DRLVM has been designed in a modular way as well:
> There is a "building engine part at the build/make and
> build/make/targets directory which is shared by all components,
> Each VM module has a building descriptor which is currently located at
> build/make/components directory, but can also be easily moved to the
> component source tree to support the idea of full independent checkout
> of a specific module.
>
> I think the building system provided with DRLVM can easily be used to
> support the source modularization approach, the proposed 'hdk' in that
> case would provide the developers, besides the "public includes", with
> the shared part of the building scripts as well.
We should continue to collaborate on finding the best solution -- it has
worked very well so far!
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Andrey Chernyshev <a....@gmail.com>.
Hi Oliver,
> I think using "src/main" and "src/test" to group our implementation
> and test code was a convention we agreed on a while back. Personally
> I dont have any problem with it, but it's something we can look at again
The current layout is just fine with me as well, in general. I just
thought that, once a big movement over a filesystem starts, it could
be a good chance to remove a few extra levels, in case we find them
redundant. If we don't think they are redundant, then let them leave
as they are.
> modules/text/src/main/native/text/
> modules/text/src/main/native/unicode/
>
> I think this agrees with what you were saying - please let me know if
> I've misunderstood!
Actually I thought of having the BidiWrapper.c, for example, directly
under the modules/text/src/main/native dir (if not considering
various OSes and platforms at this time:)). Since we already have a
'text' directory once in the beginning of the path, it may probably
look a bit excessive to repeat it once again at the end.
Thanks,
Andrey.
On 5/15/06, Oliver Deakin <ol...@googlemail.com> wrote:
> Hi Andrey,
>
>
> Andrey Chernyshev wrote:
> > On 5/12/06, Oliver Deakin <ol...@googlemail.com> wrote:
> >> Geir Magnusson Jr wrote:
> >>
> <SNIP>
> >>
> >> >>
> >> >> To do this there are at least three steps needed, as far as I can
> >> see:
> >> >>
> >> >> 1) Refactor the native code into the modular structure we currently
> >> >> have for Java code and tests. This has been spoken about before at
> >> >> [1]. The native code would be located within each module at
> >> >> modules/<module name>/src/main/native. As a starting point, can I
> >> >> propose that the natives be broken down in the following way:
> >> >>
> >> >> modules/archive/src/main/native/
> >> >> |-------archive/
> >> >> |-------zip/
> >> >>
> >> >> +------zlib/ modules/auth/src/main/native/
> >> >> +-------auth/
> >> >>
> >> >> modules/luni/src/main/native/
> >> >> |--------common/
> >> >> |--------fdlibm/
> >> >> |--------launcher/
> >> >> |--------luni/
> >> >> |--------pool/
> >> >> |--------port/
> >> >> |--------sig/
> >> >> |--------thread/
> >> >> |--------vmi/
> >> >> +-------vmls/
> >> >>
> >> >> modules/prefs/src/main/native/
> >> >> +-------prefs/
> >> >>
> >> >> modules/text/src/main/native/
> >> >> |-------text/
> >> >> +------unicode/ (comes from
> >> >> the icu4c zips stored currently in depends)
> >> >
> >> > W/o thinking too hard about it, this works for me just fine.
> >>
> >> Great - I am starting to look at how shared includes can be handled
> >> across modules (as Mark alluded to in his earlier post in this thread
> >> [1]), and at what changes will be required to split the natives into
> >> these locations. I will be taking this in small steps, trying to get the
> >> foundation and "easy" parts done first, and raising a JIRA for each step
> >> rather than in one monolithic change.
> >
> > Great! I think splitting the sources by modules at the top level
> > directory is a good idea.
> > A few questions before the big source tree reorganization starts:
> >
> >> >> modules/archive/src/main/native/
> >> >> |-------archive/
> >> >> |-------zip/
> >
> > (1) Do we need to keep the 'main' directory in every module? If we
> > need to have a distinction between tests and sources, may be just pull
> > tests one level up and have something like:
> > archive/
> > src/
> > native/
> > java/
> > tests/
> > native/
> > java/
> > I wonder if 'main' is an extra level of the directory tree we can
> > actually avoid of (lazy people don't like typing cd too much :))
>
> I think using "src/main" and "src/test" to group our implementation
> and test code was a convention we agreed on a while back. Personally
> I dont have any problem with it, but it's something we can look at again
> if people don't like it. I think that's something that would be fairly easy
> to alter once the natives are modularised, should we wish to do so.
>
>
> >
> > (2) Why do we need to have 'luni' two times in the path, e.g.
> > modules/luni/src/main/native/luni/ ? If we need to put an additional
> > stuff like 'port' to the luni module, perhaps it could be just enough
> > to put it into a subdirectory within native, e.g:
> > modules/luni/src/native/port/ ?
>
> Maybe I am missing something, but I think what you're suggesting (putting
> port etc. directly under the native directory) is the same as I laid out
> above -
> it's quite likely that my ascii diagram of the directory layout hasnt come
> across as intended, so to clarify the resulting native directories will be:
>
> modules/archive/src/main/native/archive/
> modules/archive/src/main/native/zip/
> modules/archive/src/main/native/zlib/
>
> modules/luni/src/main/native/common/
> modules/luni/src/main/native/fdlibm/
> modules/luni/src/main/native/launcher/
> modules/luni/src/main/native/luni/
> modules/luni/src/main/native/pool/
> modules/luni/src/main/native/port/
> modules/luni/src/main/native/sig/
> modules/luni/src/main/native/thread/
> modules/luni/src/main/native/vmi/
> modules/luni/src/main/native/vmls/
>
> modules/prefs/src/main/native/prefs/
>
> modules/text/src/main/native/text/
> modules/text/src/main/native/unicode/
>
> I think this agrees with what you were saying - please let me know if
> I've misunderstood!
>
> >
> >
> > BTW, I've noticed that this proposal is very similar to the DRLVM
> > source tree organization, which is like:
> > - vm
> > - include - top level include which contains h files exported by
> > various VM components;
> > - interpreter
> > - jitrino
> > - vmcore
> > ...
> > <other VM components>
> >
> > The module vmcore, for example, contains both native and java code:
> > vmcore/src/kernel_classes
> > - native
> > - javasrc
> >
> > The building system for DRLVM has been designed in a modular way as well:
> > There is a "building engine part at the build/make and
> > build/make/targets directory which is shared by all components,
> > Each VM module has a building descriptor which is currently located at
> > build/make/components directory, but can also be easily moved to the
> > component source tree to support the idea of full independent checkout
> > of a specific module.
> >
> > I think the building system provided with DRLVM can easily be used to
> > support the source modularization approach, the proposed 'hdk' in that
> > case would provide the developers, besides the "public includes", with
> > the shared part of the building scripts as well.
> >
>
> Once the initial move of native code into the modules is completed,
> any way to enhance and better modularise the build system would be
> welcome! I will take a look at the DRLVM build system to see how
> the independent modules are handled there.
>
> Regards,
> Oliver
>
> >
> > Thank you,
> > Andrey Chernyshev
> > Intel Middleware Products Division
> >
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Hi Andrey,
Andrey Chernyshev wrote:
> On 5/12/06, Oliver Deakin <ol...@googlemail.com> wrote:
>> Geir Magnusson Jr wrote:
>>
<SNIP>
>>
>> >>
>> >> To do this there are at least three steps needed, as far as I can
>> see:
>> >>
>> >> 1) Refactor the native code into the modular structure we currently
>> >> have for Java code and tests. This has been spoken about before at
>> >> [1]. The native code would be located within each module at
>> >> modules/<module name>/src/main/native. As a starting point, can I
>> >> propose that the natives be broken down in the following way:
>> >>
>> >> modules/archive/src/main/native/
>> >> |-------archive/
>> >> |-------zip/
>> >>
>> >> +------zlib/ modules/auth/src/main/native/
>> >> +-------auth/
>> >>
>> >> modules/luni/src/main/native/
>> >> |--------common/
>> >> |--------fdlibm/
>> >> |--------launcher/
>> >> |--------luni/
>> >> |--------pool/
>> >> |--------port/
>> >> |--------sig/
>> >> |--------thread/
>> >> |--------vmi/
>> >> +-------vmls/
>> >>
>> >> modules/prefs/src/main/native/
>> >> +-------prefs/
>> >>
>> >> modules/text/src/main/native/
>> >> |-------text/
>> >> +------unicode/ (comes from
>> >> the icu4c zips stored currently in depends)
>> >
>> > W/o thinking too hard about it, this works for me just fine.
>>
>> Great - I am starting to look at how shared includes can be handled
>> across modules (as Mark alluded to in his earlier post in this thread
>> [1]), and at what changes will be required to split the natives into
>> these locations. I will be taking this in small steps, trying to get the
>> foundation and "easy" parts done first, and raising a JIRA for each step
>> rather than in one monolithic change.
>
> Great! I think splitting the sources by modules at the top level
> directory is a good idea.
> A few questions before the big source tree reorganization starts:
>
>> >> modules/archive/src/main/native/
>> >> |-------archive/
>> >> |-------zip/
>
> (1) Do we need to keep the 'main' directory in every module? If we
> need to have a distinction between tests and sources, may be just pull
> tests one level up and have something like:
> archive/
> src/
> native/
> java/
> tests/
> native/
> java/
> I wonder if 'main' is an extra level of the directory tree we can
> actually avoid of (lazy people don't like typing cd too much :))
I think using "src/main" and "src/test" to group our implementation
and test code was a convention we agreed on a while back. Personally
I dont have any problem with it, but it's something we can look at again
if people don't like it. I think that's something that would be fairly easy
to alter once the natives are modularised, should we wish to do so.
>
> (2) Why do we need to have 'luni' two times in the path, e.g.
> modules/luni/src/main/native/luni/ ? If we need to put an additional
> stuff like 'port' to the luni module, perhaps it could be just enough
> to put it into a subdirectory within native, e.g:
> modules/luni/src/native/port/ ?
Maybe I am missing something, but I think what you're suggesting (putting
port etc. directly under the native directory) is the same as I laid out
above -
it's quite likely that my ascii diagram of the directory layout hasnt come
across as intended, so to clarify the resulting native directories will be:
modules/archive/src/main/native/archive/
modules/archive/src/main/native/zip/
modules/archive/src/main/native/zlib/
modules/luni/src/main/native/common/
modules/luni/src/main/native/fdlibm/
modules/luni/src/main/native/launcher/
modules/luni/src/main/native/luni/
modules/luni/src/main/native/pool/
modules/luni/src/main/native/port/
modules/luni/src/main/native/sig/
modules/luni/src/main/native/thread/
modules/luni/src/main/native/vmi/
modules/luni/src/main/native/vmls/
modules/prefs/src/main/native/prefs/
modules/text/src/main/native/text/
modules/text/src/main/native/unicode/
I think this agrees with what you were saying - please let me know if
I've misunderstood!
>
>
> BTW, I've noticed that this proposal is very similar to the DRLVM
> source tree organization, which is like:
> - vm
> - include - top level include which contains h files exported by
> various VM components;
> - interpreter
> - jitrino
> - vmcore
> ...
> <other VM components>
>
> The module vmcore, for example, contains both native and java code:
> vmcore/src/kernel_classes
> - native
> - javasrc
>
> The building system for DRLVM has been designed in a modular way as well:
> There is a "building engine part at the build/make and
> build/make/targets directory which is shared by all components,
> Each VM module has a building descriptor which is currently located at
> build/make/components directory, but can also be easily moved to the
> component source tree to support the idea of full independent checkout
> of a specific module.
>
> I think the building system provided with DRLVM can easily be used to
> support the source modularization approach, the proposed 'hdk' in that
> case would provide the developers, besides the "public includes", with
> the shared part of the building scripts as well.
>
Once the initial move of native code into the modules is completed,
any way to enhance and better modularise the build system would be
welcome! I will take a look at the DRLVM build system to see how
the independent modules are handled there.
Regards,
Oliver
>
> Thank you,
> Andrey Chernyshev
> Intel Middleware Products Division
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Andrey Chernyshev <a....@gmail.com>.
On 5/12/06, Oliver Deakin <ol...@googlemail.com> wrote:
> Geir Magnusson Jr wrote:
> >
> >
> > Oliver Deakin wrote:
> >> Geir Magnusson Jr wrote:
> >>> Mark Hindess wrote:
> >>>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
> >>>>> Mark Hindess wrote:
> >>>>>> As the Harmony Classlib grows, I think that being able to work on a
> >>>>>> single module (or some subset of the modules) will become the
> >>>>>> typical way of working for many (perhaps even most) contributors.
> >>>>> Sure, that makes sense.
> >>>>>
> >>>>>> So I think we need a plan to support this. I also think that
> >>>>>> forcing ourselves to support this mode of working sooner rather than
> >>>>>> later will help us keep from accidentally breaking the modularity in
> >>>>>> the current build/development process.
> >>>>> Can you not work on a single module now?
> >>>>
> >>>> Yes. But only by checking out everything.
> >>>
> >>> Wouldn't you do that anyway?
> >>>
> >>> I guess thats the mystery to me. I figured that someone would do
> >>> the full checkout, and go from there.
> >>
> >> I think it's good if we can cater not only for those developers who
> >> want to completely checkout classlib trunk, but also for those who
> >> want to work only on a single module. When I say "work on a single
> >> module", I mean only checking that module's directory out of SVN, and
> >> not the code for all the other modules, thereby saving download time
> >> and disk space, not to mention the subsequent reduced build times of
> >> only recompiling a single module.
> >
> > Eh. :) We have agreement in a root issue, but not as you describe.
> > IOW, I agree that it should be that after a full build, you should be
> > able to dive into a module and work and re-build in there in isolation.
> >
> > However, I have little sympathy for the reduced download time or
> > diskspace argument.
> >
> > I *do* sympathize with [psychos like] Mark who has 4 builds at once,
> > and see that it would be efficient to have multiple independent 'dev
> > subtrees' pointing at one or more pre-built trees.
>
> It seems we have agreement that being able to rebuild a single module in
> isolation is a good idea - we have different ideas about how that may
> then be used, but I think both are valid and possible once the proper
> modularisation of code and build scripts exists. I guess once the build
> system is in place, it's up to the developer how they wish to use it :)
>
> >
> >>
> >> Currently a developer cannot work on both the native and Java code
> >> for a module without checking out the whole of classlib trunk,
> >> because the native source directories and Ant build script associated
> >> with them are completely separate from the modules. IMHO it would be
> >> great if we could get the build scripts and code repository layout
> >> into a state where a developer can just checkout a module, grab an
> >> hdk from some location and start changing and rebuilding both Java
> >> and native code (and also running tests against the rebuilt libraries).
> >
> > This is mixing two issues.
> >
> > yes, I think we should put the native code into each module, so you
> > can just work in one module, both native and Java.
> >
> > Yes, I see a reason why you would want to have a pre-build snapshot to
> > use for multiple dev subtrees, for "known good" regression testing, etc.
> >
> > But I have trouble with linking them.
>
> ok, fair enough - I have mixed these two issues together. I think this
> stems from what I said above - we just have different ideas about how
> the modularisation will be used once it's in place. When I think of
> modularisation, I imagine a developer grabbing the source just for the
> module they are interested in and a snapshot build (hdk) and working
> away with it - thus I see the modularisation and the creation of an hdk
> as being linked. As you say, really they are two separate issues.
>
> >>
> >> To do this there are at least three steps needed, as far as I can see:
> >>
> >> 1) Refactor the native code into the modular structure we currently
> >> have for Java code and tests. This has been spoken about before at
> >> [1]. The native code would be located within each module at
> >> modules/<module name>/src/main/native. As a starting point, can I
> >> propose that the natives be broken down in the following way:
> >>
> >> modules/archive/src/main/native/
> >> |-------archive/
> >> |-------zip/
> >>
> >> +------zlib/ modules/auth/src/main/native/
> >> +-------auth/
> >>
> >> modules/luni/src/main/native/
> >> |--------common/
> >> |--------fdlibm/
> >> |--------launcher/
> >> |--------luni/
> >> |--------pool/
> >> |--------port/
> >> |--------sig/
> >> |--------thread/
> >> |--------vmi/
> >> +-------vmls/
> >>
> >> modules/prefs/src/main/native/
> >> +-------prefs/
> >>
> >> modules/text/src/main/native/
> >> |-------text/
> >> +------unicode/ (comes from
> >> the icu4c zips stored currently in depends)
> >
> > W/o thinking too hard about it, this works for me just fine.
>
> Great - I am starting to look at how shared includes can be handled
> across modules (as Mark alluded to in his earlier post in this thread
> [1]), and at what changes will be required to split the natives into
> these locations. I will be taking this in small steps, trying to get the
> foundation and "easy" parts done first, and raising a JIRA for each step
> rather than in one monolithic change.
Great! I think splitting the sources by modules at the top level
directory is a good idea.
A few questions before the big source tree reorganization starts:
> >> modules/archive/src/main/native/
> >> |-------archive/
> >> |-------zip/
(1) Do we need to keep the 'main' directory in every module? If we
need to have a distinction between tests and sources, may be just pull
tests one level up and have something like:
archive/
src/
native/
java/
tests/
native/
java/
I wonder if 'main' is an extra level of the directory tree we can
actually avoid of (lazy people don't like typing cd too much :))
(2) Why do we need to have 'luni' two times in the path, e.g.
modules/luni/src/main/native/luni/ ? If we need to put an additional
stuff like 'port' to the luni module, perhaps it could be just enough
to put it into a subdirectory within native, e.g:
modules/luni/src/native/port/ ?
BTW, I've noticed that this proposal is very similar to the DRLVM
source tree organization, which is like:
- vm
- include - top level include which contains h files exported by
various VM components;
- interpreter
- jitrino
- vmcore
...
<other VM components>
The module vmcore, for example, contains both native and java code:
vmcore/src/kernel_classes
- native
- javasrc
The building system for DRLVM has been designed in a modular way as well:
There is a "building engine part at the build/make and
build/make/targets directory which is shared by all components,
Each VM module has a building descriptor which is currently located at
build/make/components directory, but can also be easily moved to the
component source tree to support the idea of full independent checkout
of a specific module.
I think the building system provided with DRLVM can easily be used to
support the source modularization approach, the proposed 'hdk' in that
case would provide the developers, besides the "public includes", with
the shared part of the building scripts as well.
Thank you,
Andrey Chernyshev
Intel Middleware Products Division
>
> >
> >>
> >> Additionally, each module may have an include directory that contains
> >> header files required across its native components. Each native
> >> component would contain subdirectories for its shared and platform
> >> specific code.
> >
> > Ok, as far as I undertsand it.
> >
> >>
> >> 2) Alter the global build scripts to start creating an hdk under the
> >> deploy directory, which contains a jdk which contains a jre. As Mark
> >> described before, this hdk would probably look something like:
> >>
> >> deploy/hdk/
> >> |--------jdk/
> >> | |-------jre/ (already being created
> >> at the moment as deploy/jre)
> >> | +------include/ (already being
> >> created at the moment as deploy/include)
> >> |--------include/
> >> +-------lib/
> >>
> >> The hdk/include directory would be much like the current
> >> native-src/<platform>/include directories, and would contain a
> >> minimal set of header files that define an interface between native
> >> modules. It would be the responsibility of the individual module to
> >> copy its required headers to this directory as a prerequisite for
> >> building.
> >> The hdk/lib directory would exist on Windows platforms, and would
> >> contain .lib files for each native component.
> >>
> >> An hdk snapshot would then be a zip of the hdk directory, and should
> >> provide everything necessary for a developer to build both native and
> >> Java code against.
> >
> > Mmmmm. I'm struggling w/ having the /jdk part of it.
>
> What is it you don't like about /jdk?
>
> >
> >>
> >> 3) Modularise native and Java build scripts so that the individual
> >> module's Ant scripts are capable of building native and Java source
> >> against an hdk, rather than having its code compiled as part of a
> >> global build.
> >
> > 1) Re modularize build scripts, Yes! I thought it was this way
> > already but that it wasn't used, because that's how we added security2
> > to the build - by just invoking from the master. IMO, that's how it
> > all should have been done.
> >
> > 2) re against an HDK, fine, but never assume it's location. Make sure
> > that I can put it anywhere on a filesystem and it works. That way you
> > can have multiple dev trees pointing to it, or let me easily "A/B"
> > test by just changing a property somewhere.
> >
> >> The rebuilt libraries would be placed back into the hdk directory, so
> >> that the developers local hdk always contains their latest built code
> >> (much as the current deploy/jre does).
> >
> > I think this is a bad idea.
> >
> > First, I think we want to support an HDK being used by multiple dev
> > trees, so you don't want those trees dropping new untested into it.
> >
> > Second, think about it - do you really want to be modifying a
> > known-good snapshot with code you are debugging? How do you "reset"?
> >
> > I'd keep the HDK separate from "deploy".
> >
> > Given each module knows it's artifacts, it should be able to
> > dynamically construct the CPs and library paths as it needs to for
> > in-module testing, which is the point of this exercise...?
> >
> > geir
> Does it matter if we overlay newly built libraries onto a snapshot?
> Isn't "resetting" as simple as unzipping the snapshot archive again from
> whereever you have it stored locally?
>
> The way I see it is:
> - If you're working on an entire checkout of classlib/trunk, you don't
> really need a snapshot build, because you can rebuild the deploy
> directory from scratch. How do you "reset" in this case? If you really
> don't want the changes you've made, you can always SVN revert and rebuild ;)
> - If you're working on a single module and don't have the whole of
> classlib checked out, you need to build against a snapshot. In this case
> I think you'd want to be able to overlay your new libraries onto the
> snapshot - otherwise what other option do you have to get a complete
> build without checking out the rest of trunk?
>
> IMHO it's not a problem to overwrite the libraries with new altered ones
> - it's what we do now with the deploy directory (whether it's a snapshot
> or built from scratch).
>
> Regards,
> Oliver
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c200605091407.k49E749L028574@d06av02.portsmouth.uk.ibm.com%3e
> >
> >>
> >>
> >> Additionally, I agree with Nathan [2] that allowing a developer
> >> working in a modular way to run the entire test suite against their
> >> work-in-progress code would be a good thing, and this could be done
> >> by bundling tests and their resources in with an hdk.
> >>
> >> Regards,
> >> Oliver
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
> >>
> >> [2]
> >> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Geir Magnusson Jr wrote:
>
>
> Oliver Deakin wrote:
>> Geir Magnusson Jr wrote:
>>> Mark Hindess wrote:
>>>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>>>> Mark Hindess wrote:
>>>>>> As the Harmony Classlib grows, I think that being able to work on a
>>>>>> single module (or some subset of the modules) will become the
>>>>>> typical way of working for many (perhaps even most) contributors.
>>>>> Sure, that makes sense.
>>>>>
>>>>>> So I think we need a plan to support this. I also think that
>>>>>> forcing ourselves to support this mode of working sooner rather than
>>>>>> later will help us keep from accidentally breaking the modularity in
>>>>>> the current build/development process.
>>>>> Can you not work on a single module now?
>>>>
>>>> Yes. But only by checking out everything.
>>>
>>> Wouldn't you do that anyway?
>>>
>>> I guess thats the mystery to me. I figured that someone would do
>>> the full checkout, and go from there.
>>
>> I think it's good if we can cater not only for those developers who
>> want to completely checkout classlib trunk, but also for those who
>> want to work only on a single module. When I say "work on a single
>> module", I mean only checking that module's directory out of SVN, and
>> not the code for all the other modules, thereby saving download time
>> and disk space, not to mention the subsequent reduced build times of
>> only recompiling a single module.
>
> Eh. :) We have agreement in a root issue, but not as you describe.
> IOW, I agree that it should be that after a full build, you should be
> able to dive into a module and work and re-build in there in isolation.
>
> However, I have little sympathy for the reduced download time or
> diskspace argument.
>
> I *do* sympathize with [psychos like] Mark who has 4 builds at once,
> and see that it would be efficient to have multiple independent 'dev
> subtrees' pointing at one or more pre-built trees.
It seems we have agreement that being able to rebuild a single module in
isolation is a good idea - we have different ideas about how that may
then be used, but I think both are valid and possible once the proper
modularisation of code and build scripts exists. I guess once the build
system is in place, it's up to the developer how they wish to use it :)
>
>>
>> Currently a developer cannot work on both the native and Java code
>> for a module without checking out the whole of classlib trunk,
>> because the native source directories and Ant build script associated
>> with them are completely separate from the modules. IMHO it would be
>> great if we could get the build scripts and code repository layout
>> into a state where a developer can just checkout a module, grab an
>> hdk from some location and start changing and rebuilding both Java
>> and native code (and also running tests against the rebuilt libraries).
>
> This is mixing two issues.
>
> yes, I think we should put the native code into each module, so you
> can just work in one module, both native and Java.
>
> Yes, I see a reason why you would want to have a pre-build snapshot to
> use for multiple dev subtrees, for "known good" regression testing, etc.
>
> But I have trouble with linking them.
ok, fair enough - I have mixed these two issues together. I think this
stems from what I said above - we just have different ideas about how
the modularisation will be used once it's in place. When I think of
modularisation, I imagine a developer grabbing the source just for the
module they are interested in and a snapshot build (hdk) and working
away with it - thus I see the modularisation and the creation of an hdk
as being linked. As you say, really they are two separate issues.
>>
>> To do this there are at least three steps needed, as far as I can see:
>>
>> 1) Refactor the native code into the modular structure we currently
>> have for Java code and tests. This has been spoken about before at
>> [1]. The native code would be located within each module at
>> modules/<module name>/src/main/native. As a starting point, can I
>> propose that the natives be broken down in the following way:
>>
>> modules/archive/src/main/native/
>> |-------archive/
>> |-------zip/
>>
>> +------zlib/ modules/auth/src/main/native/
>> +-------auth/
>>
>> modules/luni/src/main/native/
>> |--------common/
>> |--------fdlibm/
>> |--------launcher/
>> |--------luni/
>> |--------pool/
>> |--------port/
>> |--------sig/
>> |--------thread/
>> |--------vmi/
>> +-------vmls/
>>
>> modules/prefs/src/main/native/
>> +-------prefs/
>>
>> modules/text/src/main/native/
>> |-------text/
>> +------unicode/ (comes from
>> the icu4c zips stored currently in depends)
>
> W/o thinking too hard about it, this works for me just fine.
Great - I am starting to look at how shared includes can be handled
across modules (as Mark alluded to in his earlier post in this thread
[1]), and at what changes will be required to split the natives into
these locations. I will be taking this in small steps, trying to get the
foundation and "easy" parts done first, and raising a JIRA for each step
rather than in one monolithic change.
>
>>
>> Additionally, each module may have an include directory that contains
>> header files required across its native components. Each native
>> component would contain subdirectories for its shared and platform
>> specific code.
>
> Ok, as far as I undertsand it.
>
>>
>> 2) Alter the global build scripts to start creating an hdk under the
>> deploy directory, which contains a jdk which contains a jre. As Mark
>> described before, this hdk would probably look something like:
>>
>> deploy/hdk/
>> |--------jdk/
>> | |-------jre/ (already being created
>> at the moment as deploy/jre)
>> | +------include/ (already being
>> created at the moment as deploy/include)
>> |--------include/
>> +-------lib/
>>
>> The hdk/include directory would be much like the current
>> native-src/<platform>/include directories, and would contain a
>> minimal set of header files that define an interface between native
>> modules. It would be the responsibility of the individual module to
>> copy its required headers to this directory as a prerequisite for
>> building.
>> The hdk/lib directory would exist on Windows platforms, and would
>> contain .lib files for each native component.
>>
>> An hdk snapshot would then be a zip of the hdk directory, and should
>> provide everything necessary for a developer to build both native and
>> Java code against.
>
> Mmmmm. I'm struggling w/ having the /jdk part of it.
What is it you don't like about /jdk?
>
>>
>> 3) Modularise native and Java build scripts so that the individual
>> module's Ant scripts are capable of building native and Java source
>> against an hdk, rather than having its code compiled as part of a
>> global build.
>
> 1) Re modularize build scripts, Yes! I thought it was this way
> already but that it wasn't used, because that's how we added security2
> to the build - by just invoking from the master. IMO, that's how it
> all should have been done.
>
> 2) re against an HDK, fine, but never assume it's location. Make sure
> that I can put it anywhere on a filesystem and it works. That way you
> can have multiple dev trees pointing to it, or let me easily "A/B"
> test by just changing a property somewhere.
>
>> The rebuilt libraries would be placed back into the hdk directory, so
>> that the developers local hdk always contains their latest built code
>> (much as the current deploy/jre does).
>
> I think this is a bad idea.
>
> First, I think we want to support an HDK being used by multiple dev
> trees, so you don't want those trees dropping new untested into it.
>
> Second, think about it - do you really want to be modifying a
> known-good snapshot with code you are debugging? How do you "reset"?
>
> I'd keep the HDK separate from "deploy".
>
> Given each module knows it's artifacts, it should be able to
> dynamically construct the CPs and library paths as it needs to for
> in-module testing, which is the point of this exercise...?
>
> geir
Does it matter if we overlay newly built libraries onto a snapshot?
Isn't "resetting" as simple as unzipping the snapshot archive again from
whereever you have it stored locally?
The way I see it is:
- If you're working on an entire checkout of classlib/trunk, you don't
really need a snapshot build, because you can rebuild the deploy
directory from scratch. How do you "reset" in this case? If you really
don't want the changes you've made, you can always SVN revert and rebuild ;)
- If you're working on a single module and don't have the whole of
classlib checked out, you need to build against a snapshot. In this case
I think you'd want to be able to overlay your new libraries onto the
snapshot - otherwise what other option do you have to get a complete
build without checking out the rest of trunk?
IMHO it's not a problem to overwrite the libraries with new altered ones
- it's what we do now with the deploy directory (whether it's a snapshot
or built from scratch).
Regards,
Oliver
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c200605091407.k49E749L028574@d06av02.portsmouth.uk.ibm.com%3e
>
>>
>>
>> Additionally, I agree with Nathan [2] that allowing a developer
>> working in a modular way to run the entire test suite against their
>> work-in-progress code would be a good thing, and this could be done
>> by bundling tests and their resources in with an hdk.
>>
>> Regards,
>> Oliver
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
>>
>> [2]
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Oliver Deakin wrote:
> Geir Magnusson Jr wrote:
>> Mark Hindess wrote:
>>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>>> Mark Hindess wrote:
>>>>> As the Harmony Classlib grows, I think that being able to work on a
>>>>> single module (or some subset of the modules) will become the
>>>>> typical way of working for many (perhaps even most) contributors.
>>>> Sure, that makes sense.
>>>>
>>>>> So I think we need a plan to support this. I also think that
>>>>> forcing ourselves to support this mode of working sooner rather than
>>>>> later will help us keep from accidentally breaking the modularity in
>>>>> the current build/development process.
>>>> Can you not work on a single module now?
>>>
>>> Yes. But only by checking out everything.
>>
>> Wouldn't you do that anyway?
>>
>> I guess thats the mystery to me. I figured that someone would do the
>> full checkout, and go from there.
>
> I think it's good if we can cater not only for those developers who want
> to completely checkout classlib trunk, but also for those who want to
> work only on a single module. When I say "work on a single module", I
> mean only checking that module's directory out of SVN, and not the code
> for all the other modules, thereby saving download time and disk space,
> not to mention the subsequent reduced build times of only recompiling a
> single module.
Eh. :) We have agreement in a root issue, but not as you describe.
IOW, I agree that it should be that after a full build, you should be
able to dive into a module and work and re-build in there in isolation.
However, I have little sympathy for the reduced download time or
diskspace argument.
I *do* sympathize with [psychos like] Mark who has 4 builds at once, and
see that it would be efficient to have multiple independent 'dev
subtrees' pointing at one or more pre-built trees.
>
> Currently a developer cannot work on both the native and Java code for a
> module without checking out the whole of classlib trunk, because the
> native source directories and Ant build script associated with them are
> completely separate from the modules. IMHO it would be great if we could
> get the build scripts and code repository layout into a state where a
> developer can just checkout a module, grab an hdk from some location and
> start changing and rebuilding both Java and native code (and also
> running tests against the rebuilt libraries).
This is mixing two issues.
yes, I think we should put the native code into each module, so you can
just work in one module, both native and Java.
Yes, I see a reason why you would want to have a pre-build snapshot to
use for multiple dev subtrees, for "known good" regression testing, etc.
But I have trouble with linking them.
>
> To do this there are at least three steps needed, as far as I can see:
>
> 1) Refactor the native code into the modular structure we currently have
> for Java code and tests. This has been spoken about before at [1]. The
> native code would be located within each module at modules/<module
> name>/src/main/native. As a starting point, can I propose that the
> natives be broken down in the following way:
>
> modules/archive/src/main/native/
> |-------archive/
> |-------zip/
>
> +------zlib/
> modules/auth/src/main/native/
> +-------auth/
>
> modules/luni/src/main/native/
> |--------common/
> |--------fdlibm/
> |--------launcher/
> |--------luni/
> |--------pool/
> |--------port/
> |--------sig/
> |--------thread/
> |--------vmi/
> +-------vmls/
>
> modules/prefs/src/main/native/
> +-------prefs/
>
> modules/text/src/main/native/
> |-------text/
> +------unicode/ (comes from the
> icu4c zips stored currently in depends)
W/o thinking too hard about it, this works for me just fine.
>
> Additionally, each module may have an include directory that contains
> header files required across its native components. Each native
> component would contain subdirectories for its shared and platform
> specific code.
Ok, as far as I undertsand it.
>
> 2) Alter the global build scripts to start creating an hdk under the
> deploy directory, which contains a jdk which contains a jre. As Mark
> described before, this hdk would probably look something like:
>
> deploy/hdk/
> |--------jdk/
> | |-------jre/ (already being created at
> the moment as deploy/jre)
> | +------include/ (already being created
> at the moment as deploy/include)
> |--------include/
> +-------lib/
>
> The hdk/include directory would be much like the current
> native-src/<platform>/include directories, and would contain a minimal
> set of header files that define an interface between native modules. It
> would be the responsibility of the individual module to copy its
> required headers to this directory as a prerequisite for building.
> The hdk/lib directory would exist on Windows platforms, and would
> contain .lib files for each native component.
>
> An hdk snapshot would then be a zip of the hdk directory, and should
> provide everything necessary for a developer to build both native and
> Java code against.
Mmmmm. I'm struggling w/ having the /jdk part of it.
>
> 3) Modularise native and Java build scripts so that the individual
> module's Ant scripts are capable of building native and Java source
> against an hdk, rather than having its code compiled as part of a global
> build.
1) Re modularize build scripts, Yes! I thought it was this way already
but that it wasn't used, because that's how we added security2 to the
build - by just invoking from the master. IMO, that's how it all should
have been done.
2) re against an HDK, fine, but never assume it's location. Make sure
that I can put it anywhere on a filesystem and it works. That way you
can have multiple dev trees pointing to it, or let me easily "A/B" test
by just changing a property somewhere.
> The rebuilt libraries would be placed back into the hdk
> directory, so that the developers local hdk always contains their latest
> built code (much as the current deploy/jre does).
I think this is a bad idea.
First, I think we want to support an HDK being used by multiple dev
trees, so you don't want those trees dropping new untested into it.
Second, think about it - do you really want to be modifying a known-good
snapshot with code you are debugging? How do you "reset"?
I'd keep the HDK separate from "deploy".
Given each module knows it's artifacts, it should be able to dynamically
construct the CPs and library paths as it needs to for in-module
testing, which is the point of this exercise...?
geir
>
>
> Additionally, I agree with Nathan [2] that allowing a developer working
> in a modular way to run the entire test suite against their
> work-in-progress code would be a good thing, and this could be done by
> bundling tests and their resources in with an hdk.
>
> Regards,
> Oliver
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
>
> [2]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Oliver Deakin <ol...@googlemail.com>.
Geir Magnusson Jr wrote:
> Mark Hindess wrote:
>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>> Mark Hindess wrote:
>>>> As the Harmony Classlib grows, I think that being able to work on a
>>>> single module (or some subset of the modules) will become the
>>>> typical way of working for many (perhaps even most) contributors.
>>> Sure, that makes sense.
>>>
>>>> So I think we need a plan to support this. I also think that
>>>> forcing ourselves to support this mode of working sooner rather than
>>>> later will help us keep from accidentally breaking the modularity in
>>>> the current build/development process.
>>> Can you not work on a single module now?
>>
>> Yes. But only by checking out everything.
>
> Wouldn't you do that anyway?
>
> I guess thats the mystery to me. I figured that someone would do the
> full checkout, and go from there.
I think it's good if we can cater not only for those developers who want
to completely checkout classlib trunk, but also for those who want to
work only on a single module. When I say "work on a single module", I
mean only checking that module's directory out of SVN, and not the code
for all the other modules, thereby saving download time and disk space,
not to mention the subsequent reduced build times of only recompiling a
single module.
Currently a developer cannot work on both the native and Java code for a
module without checking out the whole of classlib trunk, because the
native source directories and Ant build script associated with them are
completely separate from the modules. IMHO it would be great if we could
get the build scripts and code repository layout into a state where a
developer can just checkout a module, grab an hdk from some location and
start changing and rebuilding both Java and native code (and also
running tests against the rebuilt libraries).
To do this there are at least three steps needed, as far as I can see:
1) Refactor the native code into the modular structure we currently have
for Java code and tests. This has been spoken about before at [1]. The
native code would be located within each module at modules/<module
name>/src/main/native. As a starting point, can I propose that the
natives be broken down in the following way:
modules/archive/src/main/native/
|-------archive/
|-------zip/
+------zlib/
modules/auth/src/main/native/
+-------auth/
modules/luni/src/main/native/
|--------common/
|--------fdlibm/
|--------launcher/
|--------luni/
|--------pool/
|--------port/
|--------sig/
|--------thread/
|--------vmi/
+-------vmls/
modules/prefs/src/main/native/
+-------prefs/
modules/text/src/main/native/
|-------text/
+------unicode/ (comes from the
icu4c zips stored currently in depends)
Additionally, each module may have an include directory that contains
header files required across its native components. Each native
component would contain subdirectories for its shared and platform
specific code.
2) Alter the global build scripts to start creating an hdk under the
deploy directory, which contains a jdk which contains a jre. As Mark
described before, this hdk would probably look something like:
deploy/hdk/
|--------jdk/
| |-------jre/ (already being created at
the moment as deploy/jre)
| +------include/ (already being created
at the moment as deploy/include)
|--------include/
+-------lib/
The hdk/include directory would be much like the current
native-src/<platform>/include directories, and would contain a minimal
set of header files that define an interface between native modules. It
would be the responsibility of the individual module to copy its
required headers to this directory as a prerequisite for building.
The hdk/lib directory would exist on Windows platforms, and would
contain .lib files for each native component.
An hdk snapshot would then be a zip of the hdk directory, and should
provide everything necessary for a developer to build both native and
Java code against.
3) Modularise native and Java build scripts so that the individual
module's Ant scripts are capable of building native and Java source
against an hdk, rather than having its code compiled as part of a global
build. The rebuilt libraries would be placed back into the hdk
directory, so that the developers local hdk always contains their latest
built code (much as the current deploy/jre does).
Additionally, I agree with Nathan [2] that allowing a developer working
in a modular way to run the entire test suite against their
work-in-progress code would be a good thing, and this could be done by
bundling tests and their resources in with an hdk.
Regards,
Oliver
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
[2]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Vladimir Gorr <vv...@gmail.com>.
The existent snapshot is intended for this purpose. However I'm not 100%
sure it will help
for your case when the C development environment is absent. I mean the
default Windows installation
should contain some set of dynamic libaries the snapshot can loaded at
runtime.
Thanks,
Vladimir.
On 5/11/06, Damian Hamill <da...@herculeez.com> wrote:
>
> If you are working on Windows and don't have a C development environment
> setup the build fails right away. It might be useful to provide a
> mechanism for obtaining the windows binary components so people who are
> only interested in the java parts of harmony can build everything after
> a checkout.
>
> regards
> damian
>
> Geir Magnusson Jr wrote:
> > Mark Hindess wrote:
> >> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
> >>> Mark Hindess wrote:
> >>>> As the Harmony Classlib grows, I think that being able to work on a
> >>>> single module (or some subset of the modules) will become the
> >>>> typical way of working for many (perhaps even most) contributors.
> >>> Sure, that makes sense.
> >>>
> >>>> So I think we need a plan to support this. I also think that
> >>>> forcing ourselves to support this mode of working sooner rather than
> >>>> later will help us keep from accidentally breaking the modularity in
> >>>> the current build/development process.
> >>> Can you not work on a single module now?
> >>
> >> Yes. But only by checking out everything.
> >
> > Wouldn't you do that anyway?
> >
> > I guess thats the mystery to me. I figured that someone would do the
> > full checkout, and go from there.
> >
> >>
> >>>> Oliver is currently looking at restructuring the native source to
> >>>> the appropriate modules/*/src/main/native directory.
> >>> Can he talk about it out here,
> >>
> >> He/We did - more than once. For instance:
> >>
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
> >>
> >>
> >> There are quite a lot of issues when you think about how to get from
> >> where we are today to where we want to be. It's not as trivial as it
> >> sounds so even writing a good description of the issue(s) for the list
> >> is not straightforward.
> >
> > There are few alternatives though in an Apache project.
> >
> >>
> >>> or is this for your internal use?
> >>
> >> Definitely not. We sometimes talk to each other off-list but not very
> >> often! ;-)
> >>
> >>>> One question that comes out of this investigation is: where should
> >>>> the include files "live"? I think they belong with the module that
> >>>> provides the implementation defined by the declarations in the
> >>>> header. That is, zipsup.h is "owned" by archive, hythread.h is
> >>>> "owned" by thread (luni), etc.
> >>
> >>> Sure - that makes sense.
> >>
> >>>> However, if the build was to refer to the header files within the
> >>>> modules then this breaks the modularity. So, for example, someone
> >>>> working on a single module would have to checkout all the dependent
> >>>> modules rather than just the one they wish to work on.
> >>>>
> >>>> So, perhaps, at build time we should copy the header files out of
> >>>> the owning module into a common include directory. All modules
> >>>> would then refer to the header file in the common include directory.
> >>
> >> You didn't comment on this bit.
> >
> > No - I had something, and removed it so I could think about it. In
> > past lives, I've done a lot of C and C++, so was trying to recall any
> > tricks we had.
> >
> > I didn't remember any. I suppose this is an example of the downside
> > of working in glorified assembler or glorified assembler with bolt-on
> > OO extensions. :)
> >
> >>
> >> I don't think it's a good idea for one module referencing into another
> >> directly where it isn't a well-defined interface that we can
> >> manage/enforce appropriately. Would you agree?
> >
> > But I don't see a solution here, yet. Just balling things up into a
> > "HDK" doesn't seem to fix it.
> >
> >>
> >> We might as well try to agree how to construct and manage this
> >> "interface". I think the copying/"hdk" idea is a good solution.
> >
> > I guess I don't grok how the copying solves the root problem of
> > coupling, because the cross-coupling due to C/C++ is still there.
> >
> > I certainly can see that the copying is useful - for a given platform
> > target, you can copy the stuff to the top, I guess.
> >
> > But I can also see having a "top" level per module, for which the
> > target platform is copied into from the bowels of the module...
> >
> > That seems cleaner and keeps separate "namespaces".
> >
> >>
> >> I didn't imagine these issues would be this contentious or perhaps I'd
> >> have tried to separate these two (related) issues.
> >
> > That may be the problem here - I'm confusing the two issues?
> >
> >>
> >>>> This means we can then create a snapshot tar/zip of the common
> >>>> include directory which someone wishing to work on a single module
> >>>> could download to build against. This is not dissimilar to the
> >>>> current way in which you could build the java sources against a
> >>>> deploy/jre snapshot.
> >>
> >>> Why wouldn't someone wishing to work on a single module just checkout
> >>> all the code?
> >>
> >> So someone working on prefs (which is approximately 2MB) would need to
> >> check out all the source for luni (the current largest module at
> ~36MB),
> >> awt, swing, sound, etc. ?
> >
> > Yes. I actually think they would if it builds. It appears to me that
> > the classlibrary is intercoupled - our modules are an unnatural (to
> > the status quo) segmentation we've placed on top in our quest for
> > something better.
> >
> > I guess I need to re-read and figure out how this would solve that
> > problem. It seems to just add to the number of moving parts.
> >
> >>
> >> I usually have half a dozen workspaces where I'm trying things out
> (even
> >> more at the moment since I'm looking at the four contributions that are
> >> being voted on). This isn't too bad at the moment with each one being
> >> about 1/4 GB but it will get bigger over time and therefore less
> >> manageable.
> >
> > Aha...
> >>
> >>> I'm really wary about having non-SVN-sourced artifacts meant to be
> >>> used in building/development.
> >>
> >> Isn't that modularity? Why shouldn't we do it - our customers will be?
> >
> > Because our users will be using released things. Things that are
> > stable. A "*DK" by definition is stable, stable code, stable
> > interfaces, etc. (ok, slow moving, really, but you know what I mean...)
> >
> >>
> >>> Smells a bit like the first step towards the classic Windows
> >>> "dll-hell".
> >>
> >> Undoubtedly, with people working on separate modules, we will get build
> >> breaks. But we'll get them when, for example, we don't have sufficient
> >> unit test coverage within the module being worked on - we had an
> example
> >> of this not long ago IIRC. We'll also have breaks when people have
> made
> >> different assumptions about the meaning of the spec or the definition
> of
> >> the internal API. It is a good thing that we find these *bugs* within
> >> the project... being our own customers!
> >
> > I don't understand how a "hdk" helps find the bugs.
> >
> >>
> >> I think this will actually help *avoid* some of the problems you are
> >> thinking about.
> >>
> >>>> For windows, the snapshot would also include the .lib files that are
> >>>> required to build for the run-time .dll files.
> >>>>
> >>>> What is this new snapshot? Well, Tim suggested
> >>> (Where?)
> >>>
> >>>> that it was a little like the jdk but for Harmony development. So
> >>>> perhaps it should be called an hdk, Harmony Development Kit?
> >>> I'm missing the point... why wouldn't a developer checkout head, build
> >>> that, and then get to work on whatever part of whatever module they
> >>> wanted?
> >>
> >> The classlib is getting pretty big. If we are serious about
> modularity,
> >> then we should try to support it right through from build, development,
> >> deployment and ultimately at runtime.
> >
> > I think that we're serious about modularity as a packaging option (our
> > primary one) and something we're going to push for in the ecosystem.
> > However, I'm afraid of letting the tail wag the dog here...
> >
> > [SNIP]
> >
> >>
> >>> Is it that since modules reference other modules, working in one
> >>> module means you need the dependent modules around?
> >>
> >> Yes, exactly. We shouldn't require users to have the entire "source"
> >> for all the dependent modules around.
> >
> > Are you accidentally conflating "user" and "developer" here? They
> > are entirely different roles.
> >
> > Also, I certainly can see the theoretical value of this, but we tried
> > similar things in Geronimo, and it was, well, hell, because of the
> > coupling between Geronimo and it's dependencies. eventually, we gave
> > up and had an 'uberbuild', which wasn't so bad - you'd checkout
> > everything, do the master build, and then go dive into your module,
> > which is what I think talked about doing here before.
> >
> > So I guess I'm just resistive due to painful experience on this.
> >
> >
> >> We should support modularity at
> >> the development level. (Like we do already with the stubs for the
> >> luni/security kernel classes. We don't require developers to have the
> >> VM around.)
> >>
> >> When I'm developing C code I reference the libc header files but I
> don't
> >> go poking around including random headers from the libc source.
> >>
> >> So if I'm working on sql, I don't see why I shouldn't develop using
> >> header files and other well-defined parts of the API for other modules
> >> rather than having to have all of the source code checked out.
> >
> > Because I think version problems are going to spiral out of control.
> >
> >>
> >> Of course, we should still support people working by checking out
> >> everything but we shouldn't require it.
> >
> > This has to be the case. And IMO, whether you did something like "svn
> > co; build top" which fills in the "top-level header directory" (which
> > I think might be better per module), builds all jars, test jars, and
> > libs, and puts everything in place so you can go wander off and work
> > in a module, or do a lightweight checkout, stuff the hdk in a specific
> > place in your classlib/trunk tree, apart for versions, the situation
> > must be *identical*, or you are going to run into all sorts of build
> > and debugging/discussion hell.
> >
> >>
> >>> A long time ago we talked about something like this, a pre-build step
> >>> that produces a "hdk"-like set of artifacts that a developer could
> >>> then use if they were focused down inside a module.
> >>>
> >>> Is this the same thing returning for discussion?
> >>
> >> Not since I've been actively following the list but I'll dig about in
> >> the archives later.
> >>
> >>> Couldn't we just use a standard snapshot? Take the latest nightly,
> >>> drop into some magical place in the tree, and then go into your module
> >>> and work from there?
> >>
> >> Well, I was suggesting snapshots that might be:
> >>
> >> 1) hdk (inc. jdk and jre)
> >> 2) jdk (inc. jre)
> >> 3) jre
> >>
> >> but I think (?) you are suggesting using the one I overlooked:
> >>
> >> 0) everything in classlib/trunk
> >>
> >> I think 0) is going to get pretty big (but we should still create it)
> >> and I think we should actively support using 1) for development too.
> >>
> >> Wasn't someone recently pleading with Eclipse to make smaller
> artifacts?
> >> ;-)
> >
> > That's a totally different problem.
> >
> > In the end, I don't care if we snapshot things like this to make it
> > easier, but I'm really worried about what this will become.
> >
> > Also, I think it would be better to be clear about the issues in here.
> >
> > So assuming I understand the issues, I'm not against this as long as
> > the world is indistinguishable if I do a svn co and a "make the world"
> > top-level build, or just checkout a module and drop in a hdk above it
> > in the tree.
> >
> > AS a matter of fact, I think that the hdk is simple a tar of the junk
> > created by a "make the world" build...
> >
> >
> > Also, for you, with multiple workspaces, I would imagine that your
> > life would be better with this being resolvable via a "pointer" in the
> > build properties (which defaults to "." or -ish), so you can have both
> > a full tree around, as well as one or more hdk snapshots.
> >
> > The following is an example of having the full tree ('full_checkout')
> > at the same time as a hdk ('binary snapshot aka hdk') with three work
> > areas (just the prefs checked out from SVN, the RMI contribution from
> > Intel and the RMI contribution from ITC)
> >
> > /your_see_drive :)
> > /full_checkout/
> > /deploy/
> > artifacts
> > / whatever/
> > /binary snapshot aka hdk/
> > /deploy/
> > artifacts
> > /checkout_of_pref_only/
> > build.props (points to /binary snapshot aka hdk/)
> > /modules/
> > /pref/
> > /contribution_1_from_SVN/
> > build.props (points to /full checkout/)
> > /modules/
> > /RMI from Intel/
> > /contribution_2_from_SVN/
> > build.props (points to /full checkout/)
> > /modules/
> > /RMI_from_ITC/
> >
> >
> > That way, you can dork w/ the full_checkout, fix something, and then
> > your other work environments that are pointing at it get that fix w/o
> > any work.
> >
> > I hope my crude art makes sense.
> >
> > geir
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: Supporting working on a single module?
Posted by Damian Hamill <da...@herculeez.com>.
If you are working on Windows and don't have a C development environment
setup the build fails right away. It might be useful to provide a
mechanism for obtaining the windows binary components so people who are
only interested in the java parts of harmony can build everything after
a checkout.
regards
damian
Geir Magnusson Jr wrote:
> Mark Hindess wrote:
>> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>> Mark Hindess wrote:
>>>> As the Harmony Classlib grows, I think that being able to work on a
>>>> single module (or some subset of the modules) will become the
>>>> typical way of working for many (perhaps even most) contributors.
>>> Sure, that makes sense.
>>>
>>>> So I think we need a plan to support this. I also think that
>>>> forcing ourselves to support this mode of working sooner rather than
>>>> later will help us keep from accidentally breaking the modularity in
>>>> the current build/development process.
>>> Can you not work on a single module now?
>>
>> Yes. But only by checking out everything.
>
> Wouldn't you do that anyway?
>
> I guess thats the mystery to me. I figured that someone would do the
> full checkout, and go from there.
>
>>
>>>> Oliver is currently looking at restructuring the native source to
>>>> the appropriate modules/*/src/main/native directory.
>>> Can he talk about it out here,
>>
>> He/We did - more than once. For instance:
>>
>>
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
>>
>>
>> There are quite a lot of issues when you think about how to get from
>> where we are today to where we want to be. It's not as trivial as it
>> sounds so even writing a good description of the issue(s) for the list
>> is not straightforward.
>
> There are few alternatives though in an Apache project.
>
>>
>>> or is this for your internal use?
>>
>> Definitely not. We sometimes talk to each other off-list but not very
>> often! ;-)
>>
>>>> One question that comes out of this investigation is: where should
>>>> the include files "live"? I think they belong with the module that
>>>> provides the implementation defined by the declarations in the
>>>> header. That is, zipsup.h is "owned" by archive, hythread.h is
>>>> "owned" by thread (luni), etc.
>>
>>> Sure - that makes sense.
>>
>>>> However, if the build was to refer to the header files within the
>>>> modules then this breaks the modularity. So, for example, someone
>>>> working on a single module would have to checkout all the dependent
>>>> modules rather than just the one they wish to work on.
>>>>
>>>> So, perhaps, at build time we should copy the header files out of
>>>> the owning module into a common include directory. All modules
>>>> would then refer to the header file in the common include directory.
>>
>> You didn't comment on this bit.
>
> No - I had something, and removed it so I could think about it. In
> past lives, I've done a lot of C and C++, so was trying to recall any
> tricks we had.
>
> I didn't remember any. I suppose this is an example of the downside
> of working in glorified assembler or glorified assembler with bolt-on
> OO extensions. :)
>
>>
>> I don't think it's a good idea for one module referencing into another
>> directly where it isn't a well-defined interface that we can
>> manage/enforce appropriately. Would you agree?
>
> But I don't see a solution here, yet. Just balling things up into a
> "HDK" doesn't seem to fix it.
>
>>
>> We might as well try to agree how to construct and manage this
>> "interface". I think the copying/"hdk" idea is a good solution.
>
> I guess I don't grok how the copying solves the root problem of
> coupling, because the cross-coupling due to C/C++ is still there.
>
> I certainly can see that the copying is useful - for a given platform
> target, you can copy the stuff to the top, I guess.
>
> But I can also see having a "top" level per module, for which the
> target platform is copied into from the bowels of the module...
>
> That seems cleaner and keeps separate "namespaces".
>
>>
>> I didn't imagine these issues would be this contentious or perhaps I'd
>> have tried to separate these two (related) issues.
>
> That may be the problem here - I'm confusing the two issues?
>
>>
>>>> This means we can then create a snapshot tar/zip of the common
>>>> include directory which someone wishing to work on a single module
>>>> could download to build against. This is not dissimilar to the
>>>> current way in which you could build the java sources against a
>>>> deploy/jre snapshot.
>>
>>> Why wouldn't someone wishing to work on a single module just checkout
>>> all the code?
>>
>> So someone working on prefs (which is approximately 2MB) would need to
>> check out all the source for luni (the current largest module at ~36MB),
>> awt, swing, sound, etc. ?
>
> Yes. I actually think they would if it builds. It appears to me that
> the classlibrary is intercoupled - our modules are an unnatural (to
> the status quo) segmentation we've placed on top in our quest for
> something better.
>
> I guess I need to re-read and figure out how this would solve that
> problem. It seems to just add to the number of moving parts.
>
>>
>> I usually have half a dozen workspaces where I'm trying things out (even
>> more at the moment since I'm looking at the four contributions that are
>> being voted on). This isn't too bad at the moment with each one being
>> about 1/4 GB but it will get bigger over time and therefore less
>> manageable.
>
> Aha...
>>
>>> I'm really wary about having non-SVN-sourced artifacts meant to be
>>> used in building/development.
>>
>> Isn't that modularity? Why shouldn't we do it - our customers will be?
>
> Because our users will be using released things. Things that are
> stable. A "*DK" by definition is stable, stable code, stable
> interfaces, etc. (ok, slow moving, really, but you know what I mean...)
>
>>
>>> Smells a bit like the first step towards the classic Windows
>>> "dll-hell".
>>
>> Undoubtedly, with people working on separate modules, we will get build
>> breaks. But we'll get them when, for example, we don't have sufficient
>> unit test coverage within the module being worked on - we had an example
>> of this not long ago IIRC. We'll also have breaks when people have made
>> different assumptions about the meaning of the spec or the definition of
>> the internal API. It is a good thing that we find these *bugs* within
>> the project... being our own customers!
>
> I don't understand how a "hdk" helps find the bugs.
>
>>
>> I think this will actually help *avoid* some of the problems you are
>> thinking about.
>>
>>>> For windows, the snapshot would also include the .lib files that are
>>>> required to build for the run-time .dll files.
>>>>
>>>> What is this new snapshot? Well, Tim suggested
>>> (Where?)
>>>
>>>> that it was a little like the jdk but for Harmony development. So
>>>> perhaps it should be called an hdk, Harmony Development Kit?
>>> I'm missing the point... why wouldn't a developer checkout head, build
>>> that, and then get to work on whatever part of whatever module they
>>> wanted?
>>
>> The classlib is getting pretty big. If we are serious about modularity,
>> then we should try to support it right through from build, development,
>> deployment and ultimately at runtime.
>
> I think that we're serious about modularity as a packaging option (our
> primary one) and something we're going to push for in the ecosystem.
> However, I'm afraid of letting the tail wag the dog here...
>
> [SNIP]
>
>>
>>> Is it that since modules reference other modules, working in one
>>> module means you need the dependent modules around?
>>
>> Yes, exactly. We shouldn't require users to have the entire "source"
>> for all the dependent modules around.
>
> Are you accidentally conflating "user" and "developer" here? They
> are entirely different roles.
>
> Also, I certainly can see the theoretical value of this, but we tried
> similar things in Geronimo, and it was, well, hell, because of the
> coupling between Geronimo and it's dependencies. eventually, we gave
> up and had an 'uberbuild', which wasn't so bad - you'd checkout
> everything, do the master build, and then go dive into your module,
> which is what I think talked about doing here before.
>
> So I guess I'm just resistive due to painful experience on this.
>
>
>> We should support modularity at
>> the development level. (Like we do already with the stubs for the
>> luni/security kernel classes. We don't require developers to have the
>> VM around.)
>>
>> When I'm developing C code I reference the libc header files but I don't
>> go poking around including random headers from the libc source.
>>
>> So if I'm working on sql, I don't see why I shouldn't develop using
>> header files and other well-defined parts of the API for other modules
>> rather than having to have all of the source code checked out.
>
> Because I think version problems are going to spiral out of control.
>
>>
>> Of course, we should still support people working by checking out
>> everything but we shouldn't require it.
>
> This has to be the case. And IMO, whether you did something like "svn
> co; build top" which fills in the "top-level header directory" (which
> I think might be better per module), builds all jars, test jars, and
> libs, and puts everything in place so you can go wander off and work
> in a module, or do a lightweight checkout, stuff the hdk in a specific
> place in your classlib/trunk tree, apart for versions, the situation
> must be *identical*, or you are going to run into all sorts of build
> and debugging/discussion hell.
>
>>
>>> A long time ago we talked about something like this, a pre-build step
>>> that produces a "hdk"-like set of artifacts that a developer could
>>> then use if they were focused down inside a module.
>>>
>>> Is this the same thing returning for discussion?
>>
>> Not since I've been actively following the list but I'll dig about in
>> the archives later.
>>
>>> Couldn't we just use a standard snapshot? Take the latest nightly,
>>> drop into some magical place in the tree, and then go into your module
>>> and work from there?
>>
>> Well, I was suggesting snapshots that might be:
>>
>> 1) hdk (inc. jdk and jre)
>> 2) jdk (inc. jre)
>> 3) jre
>>
>> but I think (?) you are suggesting using the one I overlooked:
>>
>> 0) everything in classlib/trunk
>>
>> I think 0) is going to get pretty big (but we should still create it)
>> and I think we should actively support using 1) for development too.
>>
>> Wasn't someone recently pleading with Eclipse to make smaller artifacts?
>> ;-)
>
> That's a totally different problem.
>
> In the end, I don't care if we snapshot things like this to make it
> easier, but I'm really worried about what this will become.
>
> Also, I think it would be better to be clear about the issues in here.
>
> So assuming I understand the issues, I'm not against this as long as
> the world is indistinguishable if I do a svn co and a "make the world"
> top-level build, or just checkout a module and drop in a hdk above it
> in the tree.
>
> AS a matter of fact, I think that the hdk is simple a tar of the junk
> created by a "make the world" build...
>
>
> Also, for you, with multiple workspaces, I would imagine that your
> life would be better with this being resolvable via a "pointer" in the
> build properties (which defaults to "." or -ish), so you can have both
> a full tree around, as well as one or more hdk snapshots.
>
> The following is an example of having the full tree ('full_checkout')
> at the same time as a hdk ('binary snapshot aka hdk') with three work
> areas (just the prefs checked out from SVN, the RMI contribution from
> Intel and the RMI contribution from ITC)
>
> /your_see_drive :)
> /full_checkout/
> /deploy/
> artifacts
> / whatever/
> /binary snapshot aka hdk/
> /deploy/
> artifacts
> /checkout_of_pref_only/
> build.props (points to /binary snapshot aka hdk/)
> /modules/
> /pref/
> /contribution_1_from_SVN/
> build.props (points to /full checkout/)
> /modules/
> /RMI from Intel/
> /contribution_2_from_SVN/
> build.props (points to /full checkout/)
> /modules/
> /RMI_from_ITC/
>
>
> That way, you can dork w/ the full_checkout, fix something, and then
> your other work environments that are pointing at it get that fix w/o
> any work.
>
> I hope my crude art makes sense.
>
> geir
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Supporting working on a single module?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Mark Hindess wrote:
> On 9 May 2006 at 10:32, Geir Magnusson Jr <ge...@pobox.com> wrote:
>> Mark Hindess wrote:
>>> As the Harmony Classlib grows, I think that being able to work on a
>>> single module (or some subset of the modules) will become the
>>> typical way of working for many (perhaps even most) contributors.
>> Sure, that makes sense.
>>
>>> So I think we need a plan to support this. I also think that
>>> forcing ourselves to support this mode of working sooner rather than
>>> later will help us keep from accidentally breaking the modularity in
>>> the current build/development process.
>> Can you not work on a single module now?
>
> Yes. But only by checking out everything.
Wouldn't you do that anyway?
I guess thats the mystery to me. I figured that someone would do the
full checkout, and go from there.
>
>>> Oliver is currently looking at restructuring the native source to
>>> the appropriate modules/*/src/main/native directory.
>> Can he talk about it out here,
>
> He/We did - more than once. For instance:
>
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c4410328E.4070904@googlemail.com%3e
>
> There are quite a lot of issues when you think about how to get from
> where we are today to where we want to be. It's not as trivial as it
> sounds so even writing a good description of the issue(s) for the list
> is not straightforward.
There are few alternatives though in an Apache project.
>
>> or is this for your internal use?
>
> Definitely not. We sometimes talk to each other off-list but not very
> often! ;-)
>
>>> One question that comes out of this investigation is: where should
>>> the include files "live"? I think they belong with the module that
>>> provides the implementation defined by the declarations in the
>>> header. That is, zipsup.h is "owned" by archive, hythread.h is
>>> "owned" by thread (luni), etc.
>
>> Sure - that makes sense.
>
>>> However, if the build was to refer to the header files within the
>>> modules then this breaks the modularity. So, for example, someone
>>> working on a single module would have to checkout all the dependent
>>> modules rather than just the one they wish to work on.
>>>
>>> So, perhaps, at build time we should copy the header files out of
>>> the owning module into a common include directory. All modules
>>> would then refer to the header file in the common include directory.
>
> You didn't comment on this bit.
No - I had something, and removed it so I could think about it. In past
lives, I've done a lot of C and C++, so was trying to recall any tricks
we had.
I didn't remember any. I suppose this is an example of the downside of
working in glorified assembler or glorified assembler with bolt-on OO
extensions. :)
>
> I don't think it's a good idea for one module referencing into another
> directly where it isn't a well-defined interface that we can
> manage/enforce appropriately. Would you agree?
But I don't see a solution here, yet. Just balling things up into a
"HDK" doesn't seem to fix it.
>
> We might as well try to agree how to construct and manage this
> "interface". I think the copying/"hdk" idea is a good solution.
I guess I don't grok how the copying solves the root problem of
coupling, because the cross-coupling due to C/C++ is still there.
I certainly can see that the copying is useful - for a given platform
target, you can copy the stuff to the top, I guess.
But I can also see having a "top" level per module, for which the target
platform is copied into from the bowels of the module...
That seems cleaner and keeps separate "namespaces".
>
> I didn't imagine these issues would be this contentious or perhaps I'd
> have tried to separate these two (related) issues.
That may be the problem here - I'm confusing the two issues?
>
>>> This means we can then create a snapshot tar/zip of the common
>>> include directory which someone wishing to work on a single module
>>> could download to build against. This is not dissimilar to the
>>> current way in which you could build the java sources against a
>>> deploy/jre snapshot.
>
>> Why wouldn't someone wishing to work on a single module just checkout
>> all the code?
>
> So someone working on prefs (which is approximately 2MB) would need to
> check out all the source for luni (the current largest module at ~36MB),
> awt, swing, sound, etc. ?
Yes. I actually think they would if it builds. It appears to me that
the classlibrary is intercoupled - our modules are an unnatural (to the
status quo) segmentation we've placed on top in our quest for something
better.
I guess I need to re-read and figure out how this would solve that
problem. It seems to just add to the number of moving parts.
>
> I usually have half a dozen workspaces where I'm trying things out (even
> more at the moment since I'm looking at the four contributions that are
> being voted on). This isn't too bad at the moment with each one being
> about 1/4 GB but it will get bigger over time and therefore less
> manageable.
Aha...
>
>> I'm really wary about having non-SVN-sourced artifacts meant to be
>> used in building/development.
>
> Isn't that modularity? Why shouldn't we do it - our customers will be?
Because our users will be using released things. Things that are
stable. A "*DK" by definition is stable, stable code, stable
interfaces, etc. (ok, slow moving, really, but you know what I mean...)
>
>> Smells a bit like the first step towards the classic Windows
>> "dll-hell".
>
> Undoubtedly, with people working on separate modules, we will get build
> breaks. But we'll get them when, for example, we don't have sufficient
> unit test coverage within the module being worked on - we had an example
> of this not long ago IIRC. We'll also have breaks when people have made
> different assumptions about the meaning of the spec or the definition of
> the internal API. It is a good thing that we find these *bugs* within
> the project... being our own customers!
I don't understand how a "hdk" helps find the bugs.
>
> I think this will actually help *avoid* some of the problems you are
> thinking about.
>
>>> For windows, the snapshot would also include the .lib files that are
>>> required to build for the run-time .dll files.
>>>
>>> What is this new snapshot? Well, Tim suggested
>> (Where?)
>>
>>> that it was a little like the jdk but for Harmony development. So
>>> perhaps it should be called an hdk, Harmony Development Kit?
>> I'm missing the point... why wouldn't a developer checkout head, build
>> that, and then get to work on whatever part of whatever module they
>> wanted?
>
> The classlib is getting pretty big. If we are serious about modularity,
> then we should try to support it right through from build, development,
> deployment and ultimately at runtime.
I think that we're serious about modularity as a packaging option (our
primary one) and something we're going to push for in the ecosystem.
However, I'm afraid of letting the tail wag the dog here...
[SNIP]
>
>> Is it that since modules reference other modules, working in one
>> module means you need the dependent modules around?
>
> Yes, exactly. We shouldn't require users to have the entire "source"
> for all the dependent modules around.
Are you accidentally conflating "user" and "developer" here? They are
entirely different roles.
Also, I certainly can see the theoretical value of this, but we tried
similar things in Geronimo, and it was, well, hell, because of the
coupling between Geronimo and it's dependencies. eventually, we gave up
and had an 'uberbuild', which wasn't so bad - you'd checkout everything,
do the master build, and then go dive into your module, which is what I
think talked about doing here before.
So I guess I'm just resistive due to painful experience on this.
> We should support modularity at
> the development level. (Like we do already with the stubs for the
> luni/security kernel classes. We don't require developers to have the
> VM around.)
>
> When I'm developing C code I reference the libc header files but I don't
> go poking around including random headers from the libc source.
>
> So if I'm working on sql, I don't see why I shouldn't develop using
> header files and other well-defined parts of the API for other modules
> rather than having to have all of the source code checked out.
Because I think version problems are going to spiral out of control.
>
> Of course, we should still support people working by checking out
> everything but we shouldn't require it.
This has to be the case. And IMO, whether you did something like "svn
co; build top" which fills in the "top-level header directory" (which I
think might be better per module), builds all jars, test jars, and libs,
and puts everything in place so you can go wander off and work in a
module, or do a lightweight checkout, stuff the hdk in a specific place
in your classlib/trunk tree, apart for versions, the situation must be
*identical*, or you are going to run into all sorts of build and
debugging/discussion hell.
>
>> A long time ago we talked about something like this, a pre-build step
>> that produces a "hdk"-like set of artifacts that a developer could
>> then use if they were focused down inside a module.
>>
>> Is this the same thing returning for discussion?
>
> Not since I've been actively following the list but I'll dig about in
> the archives later.
>
>> Couldn't we just use a standard snapshot? Take the latest nightly,
>> drop into some magical place in the tree, and then go into your module
>> and work from there?
>
> Well, I was suggesting snapshots that might be:
>
> 1) hdk (inc. jdk and jre)
> 2) jdk (inc. jre)
> 3) jre
>
> but I think (?) you are suggesting using the one I overlooked:
>
> 0) everything in classlib/trunk
>
> I think 0) is going to get pretty big (but we should still create it)
> and I think we should actively support using 1) for development too.
>
> Wasn't someone recently pleading with Eclipse to make smaller artifacts?
> ;-)
That's a totally different problem.
In the end, I don't care if we snapshot things like this to make it
easier, but I'm really worried about what this will become.
Also, I think it would be better to be clear about the issues in here.
So assuming I understand the issues, I'm not against this as long as the
world is indistinguishable if I do a svn co and a "make the world"
top-level build, or just checkout a module and drop in a hdk above it in
the tree.
AS a matter of fact, I think that the hdk is simple a tar of the junk
created by a "make the world" build...
Also, for you, with multiple workspaces, I would imagine that your life
would be better with this being resolvable via a "pointer" in the build
properties (which defaults to "." or -ish), so you can have both a full
tree around, as well as one or more hdk snapshots.
The following is an example of having the full tree ('full_checkout') at
the same time as a hdk ('binary snapshot aka hdk') with three work areas
(just the prefs checked out from SVN, the RMI contribution from Intel
and the RMI contribution from ITC)
/your_see_drive :)
/full_checkout/
/deploy/
artifacts
/ whatever/
/binary snapshot aka hdk/
/deploy/
artifacts
/checkout_of_pref_only/
build.props (points to /binary snapshot aka hdk/)
/modules/
/pref/
/contribution_1_from_SVN/
build.props (points to /full checkout/)
/modules/
/RMI from Intel/
/contribution_2_from_SVN/
build.props (points to /full checkout/)
/modules/
/RMI_from_ITC/
That way, you can dork w/ the full_checkout, fix something, and then
your other work environments that are pointing at it get that fix w/o
any work.
I hope my crude art makes sense.
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org