You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by "Geir Magnusson Jr." <ge...@pobox.com> on 2006/10/31 13:35:40 UTC
Re: [classlib] Preprocessor - CHECKPOINT
So we all agree that it's not an ideal solution.
Can anyone think of anything else? No one said this was going to be easy...
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Sian January wrote:
> On 02/11/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>>
>>
>> Sian January wrote:
>> > I believe that in CVS when you make a branch there's nothing in it to
>> > begin with, so if you check out code from the branch it looks the same
>> > as code in head. Then if a change is made in a file in head and that
>> > file hasn't been changed in the branch that change is reflected in the
>> > branch. My understanding is that SVN is based on CVS so I think it
>> > should work the same way.
>>
>> Nope.
>>
>> There is really no such thing as a branch or tag in SVN. It's just a
>> copy. I think it's the feature I miss most from CVS.
>
>
> That's very unfortunate. I had assumed that SVN was a superset of CVS as
> it's more recent, but obviously I was wrong :-(
it's a shame. I really, really miss that feature.
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Sian January <si...@googlemail.com>.
On 02/11/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
>
> Sian January wrote:
> > I believe that in CVS when you make a branch there's nothing in it to
> > begin with, so if you check out code from the branch it looks the same
> > as code in head. Then if a change is made in a file in head and that
> > file hasn't been changed in the branch that change is reflected in the
> > branch. My understanding is that SVN is based on CVS so I think it
> > should work the same way.
>
> Nope.
>
> There is really no such thing as a branch or tag in SVN. It's just a
> copy. I think it's the feature I miss most from CVS.
That's very unfortunate. I had assumed that SVN was a superset of CVS as
it's more recent, but obviously I was wrong :-(
> So my point was that if it's only a small
> > number of classes that are branched then integrating fixes shouldn't be
> > that problematic. Please feel free to correct me if any of those
> > assumptions are wrong.
> >
> > Just thinking about J2ME, I can imagine that some source files are going
> > to be very different. For example there are no Java 5 features in J2ME,
> > so any generic classes will have to be almost completely different. My
> > concern is that trying to combine two quite different classes in the
> > same file is going be very difficult to read and understand.
>
> Right. I don't know the first thing about ME. Maybe we should come up
> w/ use cases based on Java 6 (as we're going to do it at some point) and
> work from there.
Yes - that seems to make much more sense.
geir
>
> >
> > Thanks,
> >
> > Sian
> >
> >
> > On 02/11/06, *Geir Magnusson Jr.* <geir@pobox.com
> > <ma...@pobox.com>> wrote:
> >
> >
> >
> > Sian January wrote:
> > > I may be totally off track here, but how about just having two
> copies
> > > of all the files that differ? I don't believe it would be that
> many,
> > > and it would save us from having complicated source files or
> > having to
> > > use special tools or special IDE plug-ins. For me the value of
> > having
> > > clearly readable source code and being able to use an IDE out of
> > the box
> > > outweighs any extra effort there may be with this solution.
> >
> > Because I think that still means we have separate branches, and thus
> > the
> > integration problem for fixes.
> >
> > geir
> >
> > >
> > > Regards,
> > >
> > > Sian
> > >
> > >
> > >
> > > On 31/10/06, *Geir Magnusson Jr.* <geir@pobox.com
> > <ma...@pobox.com>
> > > <mailto:geir@pobox.com <ma...@pobox.com>>> wrote:
> > >
> > > So we all agree that it's not an ideal solution.
> > >
> > > Can anyone think of anything else? No one said this was
> > going to be
> > > easy...
> > >
> > > geir
> > >
> > >
> > >
> > >
> > > --
> > > Sian January
> > >
> > > IBM Java Technology Centre, UK
> >
> >
> >
> >
> > --
> > Sian January
> >
> > IBM Java Technology Centre, UK
>
--
Sian January
IBM Java Technology Centre, UK
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Sian January wrote:
> I believe that in CVS when you make a branch there's nothing in it to
> begin with, so if you check out code from the branch it looks the same
> as code in head. Then if a change is made in a file in head and that
> file hasn't been changed in the branch that change is reflected in the
> branch. My understanding is that SVN is based on CVS so I think it
> should work the same way.
Nope.
There is really no such thing as a branch or tag in SVN. It's just a
copy. I think it's the feature I miss most from CVS.
> So my point was that if it's only a small
> number of classes that are branched then integrating fixes shouldn't be
> that problematic. Please feel free to correct me if any of those
> assumptions are wrong.
>
> Just thinking about J2ME, I can imagine that some source files are going
> to be very different. For example there are no Java 5 features in J2ME,
> so any generic classes will have to be almost completely different. My
> concern is that trying to combine two quite different classes in the
> same file is going be very difficult to read and understand.
Right. I don't know the first thing about ME. Maybe we should come up
w/ use cases based on Java 6 (as we're going to do it at some point) and
work from there.
geir
>
> Thanks,
>
> Sian
>
>
> On 02/11/06, *Geir Magnusson Jr.* <geir@pobox.com
> <ma...@pobox.com>> wrote:
>
>
>
> Sian January wrote:
> > I may be totally off track here, but how about just having two copies
> > of all the files that differ? I don't believe it would be that many,
> > and it would save us from having complicated source files or
> having to
> > use special tools or special IDE plug-ins. For me the value of
> having
> > clearly readable source code and being able to use an IDE out of
> the box
> > outweighs any extra effort there may be with this solution.
>
> Because I think that still means we have separate branches, and thus
> the
> integration problem for fixes.
>
> geir
>
> >
> > Regards,
> >
> > Sian
> >
> >
> >
> > On 31/10/06, *Geir Magnusson Jr.* <geir@pobox.com
> <ma...@pobox.com>
> > <mailto:geir@pobox.com <ma...@pobox.com>>> wrote:
> >
> > So we all agree that it's not an ideal solution.
> >
> > Can anyone think of anything else? No one said this was
> going to be
> > easy...
> >
> > geir
> >
> >
> >
> >
> > --
> > Sian January
> >
> > IBM Java Technology Centre, UK
>
>
>
>
> --
> Sian January
>
> IBM Java Technology Centre, UK
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Mikhail Fursov <mi...@gmail.com>.
On 11/3/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>
> Mikhail Fursov wrote:
> > Good point! I think Java1.2 classes format is a reason to move J2ME
> > development into separate branch.
> > The only usecase for preprocessor is left (correct me if I miss
> something)
> > is Java6 changes.
> > ?
>
> a) This is "class format"... AFAIK, you can use generics for
> development, but compile with -target=1.2 or something like it, no?
>
> b) If I'm wrong in a), then what if you wanted additional branches of
> harmony: j2se1.4, j2me (classformat1.2)? Then using the proposed tool
> would be IMO preferable to using completely distinct svn branches for
> these.
>
>
> Etienne
>
>
Yes,
it's possible to write J2ME applications using Java5 syntax with
generics and use
special compiler to compile it to Java1.2 format.
My position here is:
1) Let's ask volunteers/gurus who do JavaXYZ port what is more convenient
for them
2) The idea with Preprocessor (I mean your version with a special 2-way
tool) is very good and scalable until you do not have diffs in almost every
line.
3) Today we have a lot of changes in classlib: ~100 commits per day. Merging
of branches is almost unreal. Maybe in the future situation will change
because we will have more stable
code. Maybe not: we can have 10 times more developers! Let's wait and
will see :)
--
Mikhail Fursov
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Mikhail Fursov wrote:
> Good point! I think Java1.2 classes format is a reason to move J2ME
> development into separate branch.
> The only usecase for preprocessor is left (correct me if I miss something)
> is Java6 changes.
> ?
a) This is "class format"... AFAIK, you can use generics for
development, but compile with -target=1.2 or something like it, no?
b) If I'm wrong in a), then what if you wanted additional branches of
harmony: j2se1.4, j2me (classformat1.2)? Then using the proposed tool
would be IMO preferable to using completely distinct svn branches for these.
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Mikhail Fursov <mi...@gmail.com>.
On 11/2/06, Sian January <si...@googlemail.com> wrote:
>
> Just thinking about J2ME, I can imagine that some source files are going
> to be very different. For example there are no Java 5 features in J2ME,
> so
> any generic classes will have to be almost completely different. My
> concern
> is that trying to combine two quite different classes in the same file is
> going be very difficult to read and understand.
>
Good point! I think Java1.2 classes format is a reason to move J2ME
development into separate branch.
The only usecase for preprocessor is left (correct me if I miss something)
is Java6 changes.
?
--
Mikhail Fursov
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Sian January <si...@googlemail.com>.
I believe that in CVS when you make a branch there's nothing in it to begin
with, so if you check out code from the branch it looks the same as code in
head. Then if a change is made in a file in head and that file hasn't been
changed in the branch that change is reflected in the branch. My
understanding is that SVN is based on CVS so I think it should work the same
way. So my point was that if it's only a small number of classes that
are branched then integrating fixes shouldn't be that problematic. Please
feel free to correct me if any of those assumptions are wrong.
Just thinking about J2ME, I can imagine that some source files are going
to be very different. For example there are no Java 5 features in J2ME, so
any generic classes will have to be almost completely different. My concern
is that trying to combine two quite different classes in the same file is
going be very difficult to read and understand.
Thanks,
Sian
On 02/11/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
>
> Sian January wrote:
> > I may be totally off track here, but how about just having two copies
> > of all the files that differ? I don't believe it would be that many,
> > and it would save us from having complicated source files or having to
> > use special tools or special IDE plug-ins. For me the value of having
> > clearly readable source code and being able to use an IDE out of the box
> > outweighs any extra effort there may be with this solution.
>
> Because I think that still means we have separate branches, and thus the
> integration problem for fixes.
>
> geir
>
> >
> > Regards,
> >
> > Sian
> >
> >
> >
> > On 31/10/06, *Geir Magnusson Jr.* <geir@pobox.com
> > <ma...@pobox.com>> wrote:
> >
> > So we all agree that it's not an ideal solution.
> >
> > Can anyone think of anything else? No one said this was going to be
> > easy...
> >
> > geir
> >
> >
> >
> >
> > --
> > Sian January
> >
> > IBM Java Technology Centre, UK
>
--
Sian January
IBM Java Technology Centre, UK
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Mikhail Fursov <mi...@gmail.com>.
On 11/1/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>
> Comfortable development happens only using "development targets". E.g.
>
> 1- process(X, devtarget) -> Z
>
> 2- edit Z in IDE using comfortable development, where you see a single
> commented line for every hidden stream code chunk, keeping you aware
> that other streams have related code there [you click on the "+" in
> Eclipse if you want to see the complete chunk]. Of course, you
> should never delete a chunk without consulting other stream
> developers first.
> So: edit Z -> Z'
>
> 3- revert(Z') -> X' this works, as long as devtarget is a stream code
> preserving target (a "development target").
>
> 4- svn ci of X' :-)
Etienne, thanks! Now I understand how it works.
Having such a tool seems like a very promising good idea!
--
Mikhail Fursov
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Mikhail Fursov wrote:
>> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
...
> Just my $0.02: IMO it's unreal to support J2SE 1.4 & 1.5 in the same
> source.
> Too many differences in the language due to generics. This example needs
> branches & weekly manual merges (not a big problem imho)
You're absolutely right! I wasn't thinking about it when I took that
example. This is really a typical case (j2se 1.4) where using svn
branches is the right solution.
But, for j2me1.5, java for credit cards 1.5 and j2se1.6 (maybe, if they
don't redesign the Java syntax, etc.), I think that a tool along the
lines I was proposing would be best.
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Mikhail Fursov <mi...@gmail.com>.
On 11/1/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>
> Problem
> -------
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
> Solution
> --------
>
> This is a trickier problem. We can divide the problem at two main levels:
>
> 1- file/directory level
> 2- source-code level
>
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version. In other words, j2me might
> not contain database APIs.
>
> Managing file inclusion/exclusion can be done in various ways.
Just my $0.02: IMO it's unreal to support J2SE 1.4 & 1.5 in the same source.
Too many differences in the language due to generics. This example needs
branches & weekly manual merges (not a big problem imho)
--
Mikhail Fursov
Re: [classlib] Preprocessor - out of the abstractness came forth concreteness
Posted by Alexey Petrenko <al...@gmail.com>.
Some concreteness would be great :)
2006/11/3, Tim Ellison <t....@gmail.com>:
> Geir Magnusson Jr. wrote:
> > As fun as this discussion is, it's all theoretical.
>
> There's no harm in a bit of fun :-)
>
> So how about we create a sandbox in SVN and copy over a few types to
> mark-up while Etienne is developing the processor, and we can all watch
> that evolve?
>
> The sandbox ensures that we don't disturb people who are working in
> TRUNK (and therefore not having fun ;-p) and we should plan now to throw
> away the sandbox and do the mark-up again once we settle on a workable
> solution, i.e. we'd let the sandbox code get out of date.
>
> I'm inclined to say that we tackle the 1.5/1.6 differences as our
> example. Marking-up a module like NIO would be easy enough.
>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>>
>> Because I can imagine that as a SE programmer, I would want 1 of 2 things :
>>
>> 1) clean code to work on for a specific target (say SE 5)
>>
>> 2) a hybrid version of your dev1 that only has the glop for a subset of
>> the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus
>> across just those two.
>>
>> Maybe it's just me, but I'd probably want the distractions of other
>> platforms out of the code so I can get better into the 'flow' when
>> developing.
>>
>> So that's why I think of dev1/2/3 as being the virtual code (not key to
>> the process) that can be generated either by the tooling onto disk as a
>> temp form, or by IDE virtually in situ.
>
> but how do you resolve the ambiguity of where new/moved code should go
> in the master if you are working in a dev that has lost the context of
> other streams?
I want to make sure I understand what you are asking - I think you are
asking the question "As a developer, if I can't see the content of the
other streams, how can I place the changes in context of the other streams?"
It's not clear to me that there's a big problem here - the fact that a
complete master exists means that when you "commit" your changes to the
master, you can resolve ambiguities in placement relative to other
streams at that time, *if* there are any. (For the life of me, I can't
figure out how to mechanically characterize an ambiguity...)
The big question here is if that's a reasonable approach, and the answer
will be based on how often it happens.
If it happens in 1 in 50 "commits", I'd argue that the benefit of the
reduction of clutter is worth it. If it happens every time, it's
clearly going to be a royal PITA. I don't know actually where to draw
that line. I expect testing with toy examples will help us figure it out.
>
> it's the point that was made here:
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200611.mbox/%3c454A4185.6070700@sablevm.org%3e
>
I went back and review that, and I keep getting this nagging feeling
that we're being a bit naive about this, but I can't put my finger on it
and explain why :) I'll try to explain in a separate post.
> it would be cool if we could do that.
Yes - my goal here is to make this as transparent as possible to the
working harmony developer.
>
>> Further, there aren't just N dev forms for the N distinct
>> platforms/version, but rather the freedom for combinations where they
>> make sense (a dev w/ SE 6 and SE 6 only)
>>
>> If you want to generate a dev1 and work on that on disk, great - do it,
>> you can go back to master.
>
> Only if the reverse mapping is exact, which I believe means that you
> need to take the full master context along with you.
I don't believe so. I need to do some tests. The reason I don't believe
so simply because the master context exists in the master (by defn),
which always is a part of the "commit" process - IOW, when committing a
dev1' (modified dev1), you can regenerate the clean original dev1 from
master, then find your diffs to that from dev1' (the modified thing you
are committing), and map those back into the master.
I think.
[SNIP]
>>
>> Or in simpler terms, the model differences seem to boil down to this.
>> Given N possible targets:
>>
>> 1) Master is concrete, dev1..M (where M possibly > N) are generated
>> simply for ease of use either as concrete tree on disk or virtually in IDE
>>
>> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
>> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
>> necessary? Nice, yes, but necessary?)
>
> Yes, that seems to be the difference. I'd like to be convinced that
> working in dev1..M code is possible in approach (1).
Yes, and I can almost prove now that not having a master is unworkable,
because we're not talking only about file content, but different files,
and different directory structure. So if the classlib code spans both
versions (SE 5, SE 6) as well as platforms (ME, SE), the build process
will require processing to deal with that during the build. if that's
the case, then you are in case 1), not 2).
I'm going to try to cough up some simple use-cases of code trees and
toss in the sandbox.
geir
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> One copy (the master) on which the IDE and cmd line tooling does both
> the "file/directory" masking (dealing with the differences that aren't
> just in class files - add and remove classes, etc - the "lookaside
> table") as well as code xform.
>
> I never thought of having that intermediate form (dev1/2/3) as being
> fundamental to the model, although it's a nice option. I can see how it
> will be nice for editing, but I'm instinctively suspicious of it being
> necessary).
>
> Why?
>
> Because I can imagine that as a SE programmer, I would want 1 of 2 things :
>
> 1) clean code to work on for a specific target (say SE 5)
>
> 2) a hybrid version of your dev1 that only has the glop for a subset of
> the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus
> across just those two.
>
> Maybe it's just me, but I'd probably want the distractions of other
> platforms out of the code so I can get better into the 'flow' when
> developing.
>
> So that's why I think of dev1/2/3 as being the virtual code (not key to
> the process) that can be generated either by the tooling onto disk as a
> temp form, or by IDE virtually in situ.
but how do you resolve the ambiguity of where new/moved code should go
in the master if you are working in a dev that has lost the context of
other streams?
it's the point that was made here:
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200611.mbox/%3c454A4185.6070700@sablevm.org%3e
it would be cool if we could do that.
> Further, there aren't just N dev forms for the N distinct
> platforms/version, but rather the freedom for combinations where they
> make sense (a dev w/ SE 6 and SE 6 only)
>
> If you want to generate a dev1 and work on that on disk, great - do it,
> you can go back to master.
Only if the reverse mapping is exact, which I believe means that you
need to take the full master context along with you.
> If you want to just work on the single master tree, you can do that if
> the IDE has the tooling.
>
>>
>> We also talked about how patches to the source that Mrs CLDC developer
>> sees can be matched back to a devtarget.
>
> Yes, I did read it :)
I was changing focus, you knew that <grr>
> Thanks for the example. I guess the difference is that I virtualized
> the dev1/2/3 form and used tooling to just let people work on master
> directly, or let dev1/2/3 be concrete as a temporary form for
> convenience, as I noted above.
>
> Or in simpler terms, the model differences seem to boil down to this.
> Given N possible targets:
>
> 1) Master is concrete, dev1..M (where M possibly > N) are generated
> simply for ease of use either as concrete tree on disk or virtually in IDE
>
> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
> necessary? Nice, yes, but necessary?)
Yes, that seems to be the difference. I'd like to be convinced that
working in dev1..M code is possible in approach (1).
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Tim Ellison wrote:
>> I'll wait until you have a simple contrived example working and settled
>> upon some syntax for mark-up, then I'll help to mark-up some real
>> example and test the tooling. Let me know when you have something ready
>> to try.
>
> Yes, I have to figure out some sane syntax for all this. I'll try to
> find some easily extensible syntax, so that we don't have to break
> backward compatibility every time we extend it. :-)
it may help to keep that discussion here, so anyone with ideas can help...
geir
>
> Etienne
>
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by Etienne Gagnon <eg...@sablevm.org>.
Tim Ellison wrote:
> I'll wait until you have a simple contrived example working and settled
> upon some syntax for mark-up, then I'll help to mark-up some real
> example and test the tooling. Let me know when you have something ready
> to try.
Yes, I have to figure out some sane syntax for all this. I'll try to
find some easily extensible syntax, so that we don't have to break
backward compatibility every time we extend it. :-)
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Tim Ellison wrote:
>>>> I'm inclined to say that we tackle the 1.5/1.6 differences as our
>>>> example. Marking-up a module like NIO would be easy enough.
>>> That's what I was hoping for. One concrete example :)
>>>
>>> I'm thrilled to see something happen in the sandbox (that's really the
>>> only way to do it... I'd be -1 to trying anything in the trunk. I hope
>>> that you knew that...)
>> Yes, of course. But I'll wait for Etienne to agree to the idea since
>> he's signing up to do the hard work. I'll help to mark up the classlibs.
>
> Of course, I agree!
I'll wait until you have a simple contrived example working and settled
upon some syntax for mark-up, then I'll help to mark-up some real
example and test the tooling. Let me know when you have something ready
to try.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by Etienne Gagnon <eg...@sablevm.org>.
Tim Ellison wrote:
>>>I'm inclined to say that we tackle the 1.5/1.6 differences as our
>>>example. Marking-up a module like NIO would be easy enough.
>>
>>That's what I was hoping for. One concrete example :)
>>
>>I'm thrilled to see something happen in the sandbox (that's really the
>>only way to do it... I'd be -1 to trying anything in the trunk. I hope
>>that you knew that...)
>
> Yes, of course. But I'll wait for Etienne to agree to the idea since
> he's signing up to do the hard work. I'll help to mark up the classlibs.
Of course, I agree!
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> Tim Ellison wrote:
>> I'm inclined to say that we tackle the 1.5/1.6 differences as our
>> example. Marking-up a module like NIO would be easy enough.
>
> That's what I was hoping for. One concrete example :)
>
> I'm thrilled to see something happen in the sandbox (that's really the
> only way to do it... I'd be -1 to trying anything in the trunk. I hope
> that you knew that...)
Yes, of course. But I'll wait for Etienne to agree to the idea since
he's signing up to do the hard work. I'll help to mark up the classlibs.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> As fun as this discussion is, it's all theoretical.
>
> There's no harm in a bit of fun :-)
>
> So how about we create a sandbox in SVN and copy over a few types to
> mark-up while Etienne is developing the processor, and we can all watch
> that evolve?
>
> The sandbox ensures that we don't disturb people who are working in
> TRUNK (and therefore not having fun ;-p) and we should plan now to throw
> away the sandbox and do the mark-up again once we settle on a workable
> solution, i.e. we'd let the sandbox code get out of date.
>
> I'm inclined to say that we tackle the 1.5/1.6 differences as our
> example. Marking-up a module like NIO would be easy enough.
That's what I was hoping for. One concrete example :)
I'm thrilled to see something happen in the sandbox (that's really the
only way to do it... I'd be -1 to trying anything in the trunk. I hope
that you knew that...)
geir
Re: [classlib] Preprocessor - out of the abstractness came forth
concreteness
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> As fun as this discussion is, it's all theoretical.
There's no harm in a bit of fun :-)
So how about we create a sandbox in SVN and copy over a few types to
mark-up while Etienne is developing the processor, and we can all watch
that evolve?
The sandbox ensures that we don't disturb people who are working in
TRUNK (and therefore not having fun ;-p) and we should plan now to throw
away the sandbox and do the mark-up again once we settle on a workable
solution, i.e. we'd let the sandbox code get out of date.
I'm inclined to say that we tackle the 1.5/1.6 differences as our
example. Marking-up a module like NIO would be easy enough.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> [...]
>> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
>> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
>> necessary? Nice, yes, but necessary?)
>
> I definitely think that it is necessary (where J2SE5 is the preferred
> devtarget), at least during the initial transition to using the
> processing tool.
:)
That still makes it "nice" - because you can always transform the SVN
tree to any platform anyway.
>
> It's (I think) the least intrusive method for starting to slowly add
> non-J2SE5 code in the development trunk while letting existing J2SE5
> developers continue developing without changing the way they've been
> using up to now.
Yep. "nice".
>
> By leaving the preferred devtarget in svn, the hordes of existing J2SE
> developers will have the choice of continuing to use simple "svn co/ci"
> to work on the code and won't have to learn to use new, yet not fully
> tested tools.
Yep. "nice" :)
>
> It also lets the smaller number of non-J2SE5 developers test the
> processing tools and help identify bugs (what!? :-P) and usability
> problems without forcing J2SE developers into doing the same.
>
> Maybe, in the longer term, it would make sense to have some kind of
> canonical "master" form, and store that in svn, but I am a big fan of
> the least disturbance path, when trying to put in place new processes in
> an ongoing development project. :-)
>
> Question: What would be the advantage (in the longer term) of putting,
> in the Subversion trunk, source code in non-compilable canonical master
> form, instead of compilable preferred devtarget form? Why is the
> existence of "master" so important to you?
I need to go check if where I said it wasn't to be compilable, as I
don't believe I ever mandated that. I do think it's going to be
difficult to achieve though.
I think we both agree that we need a complete, fixed reference point to
put in SVN. (In that sense, you actually have a master too, actually)
However, it's not clear to me that a master can actually be created that
is complete for all platforms while still buildable directly w/o
transforms and look-aside hiding/moving as part of the build process.
If you need to transform and move things around while building, then you
don't have the model you are advocating. You actually have the concrete
master model. You could cheat, and have both the concrete master tree
and a preferred platform transform checked in at the same time, but
that's just the concrete master model with a convenient "generated
artifact" checkin as well. That's fine, as long as you remember to
update both and checkin both on modification. We do that for the
website, for example.
Even if we are astoundingly lucky, and we can create a simple buildable
tree that can magically be transformed across platforms, I think that
not distinguishing a master form as distinct from other forms is a
mistake, because for the case of wanting a subset of representation for
clarity in development (which I'm not sure you acknowledge yet is
important), transforms are irreversible w/o a fixed master :
masterToDev(master, 2+3, clean) -> dev2+3
meaning, transform master to contain code for platforms 2 and 3 only (ex
Java SE 5 and 6), cleaning out all other, compilable for 2 but including
3 material in comment form.
Then that is no longer a peer to your dev1/2/3 set (where all info is in
all forms, just compilable on different platforms) but different because
data for all platforms except 2 and 3 were taken out.
Therefore, I do think we need an explicit master copy, with the dev
forms as intermediate convenience in whatever forms the developers
choose to put them in (with all other stuff, subset of other stuff, w/o
any other stuff...)
As fun as this discussion is, it's all theoretical. I've never worked
with multi-platform java code. I've worked with a lot of multi-platform
C code, and having the ability to ditch the confusing #ifdef cruft for
the platforms I'm not working on at the moment would be a very
innovative and pleasant advancement. Anyway, this is theory. The real
test will be how it actually evolves in code...
geir
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
> [...]
> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
> necessary? Nice, yes, but necessary?)
I definitely think that it is necessary (where J2SE5 is the preferred
devtarget), at least during the initial transition to using the
processing tool.
It's (I think) the least intrusive method for starting to slowly add
non-J2SE5 code in the development trunk while letting existing J2SE5
developers continue developing without changing the way they've been
using up to now.
By leaving the preferred devtarget in svn, the hordes of existing J2SE
developers will have the choice of continuing to use simple "svn co/ci"
to work on the code and won't have to learn to use new, yet not fully
tested tools.
It also lets the smaller number of non-J2SE5 developers test the
processing tools and help identify bugs (what!? :-P) and usability
problems without forcing J2SE developers into doing the same.
Maybe, in the longer term, it would make sense to have some kind of
canonical "master" form, and store that in svn, but I am a big fan of
the least disturbance path, when trying to put in place new processes in
an ongoing development project. :-)
Question: What would be the advantage (in the longer term) of putting,
in the Subversion trunk, source code in non-compilable canonical master
form, instead of compilable preferred devtarget form? Why is the
existence of "master" so important to you?
Etienne
PS: Tim, Thanks so much for providing clear answers and examples
explaining the approach!
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Tim Ellison wrote:
>>> There are multiple 'devtarget's -- each contains all the source code
>>> marked-up for every target being developed, but each is distinct by
>>> having different targets uncommented. Since each devtarget contains the
>>> entire source code it is possible to accurately transform from any one
>>> devtarget to any other devtarget. One of the devtargets is actually
>>> stored in svn, and that is the canonical form of the devtarget code.
>> So that's 'master'. and everythign else is a transformed master, right?
>
> Yes, but some transforms preserve all mark-up and are used to switch the
> compilable form between different targets (these are the 'devtarget's);
> and some transforms are mark-up-lossy and are used to provide the end
> user compilable form (this is the 'releasetarget').
Yes, I figured that out last night when I was driving to dinner. Let me
see if I have it right :
(Master)
/ | \
/ | \
/ | \
dev1 <-> dev2 <->dev3
| | |
targ1 targ2 targ3
where in your case, master doesn't exist (consider it virtual), dev1/2/3
are compilable under whatever platform they are targeted to yet contain
all the stuff for other platforms, and targ1/2/3 are 'clean code' code
the target platform.
There exists a transform process
interDev(dev1, 2) -> dev2
which is really conceptually
masterToDev(devToMaster(dev1), 2)
(if you had a master) and further
toCleanTarget(dev1) - > targ1
('normal' code for target platform)
>
>>> Screech -- the releasetarget, by definition, does not contain the source
>>> for all possible targets so that it is consumable by end-users.
>> That is correct.
>>
>> The only code that contains all code is master.
>
> Not in the technique that I'm talking about. All devtargets contain all
> the code (but they differ in which code is compilable and which is in
> comments).
Right - because you don't have a master
>
> Let me try using very simple example for my favourite type, assuming we
> were doing CLDC and SE1.5 development simultaneously.
[SNIP]
Thanks for the very nice example. That example conforms w/ my picture
of how things might work (although I suspect things could get *very*
ugly to work with in the intermediate dev1/2/3 form, but I think that
some real use cases will tell us that)
>
>
> We then talked about how the different devtargets can be stored in
> different working directories for use by Harmony developers with
> file-based tools, or (thinking about it for a second now) it could be an
> on the fly processing to/from canonical form on disk by the IDE editor.
Yes - that latter part is exactly what I've been advocating :
One copy (the master) on which the IDE and cmd line tooling does both
the "file/directory" masking (dealing with the differences that aren't
just in class files - add and remove classes, etc - the "lookaside
table") as well as code xform.
I never thought of having that intermediate form (dev1/2/3) as being
fundamental to the model, although it's a nice option. I can see how it
will be nice for editing, but I'm instinctively suspicious of it being
necessary).
Why?
Because I can imagine that as a SE programmer, I would want 1 of 2 things :
1) clean code to work on for a specific target (say SE 5)
2) a hybrid version of your dev1 that only has the glop for a subset of
the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus
across just those two.
Maybe it's just me, but I'd probably want the distractions of other
platforms out of the code so I can get better into the 'flow' when
developing.
So that's why I think of dev1/2/3 as being the virtual code (not key to
the process) that can be generated either by the tooling onto disk as a
temp form, or by IDE virtually in situ.
Further, there aren't just N dev forms for the N distinct
platforms/version, but rather the freedom for combinations where they
make sense (a dev w/ SE 6 and SE 6 only)
If you want to generate a dev1 and work on that on disk, great - do it,
you can go back to master.
If you want to just work on the single master tree, you can do that if
the IDE has the tooling.
>
> We also talked about how patches to the source that Mrs CLDC developer
> sees can be matched back to a devtarget.
Yes, I did read it :)
Thanks for the example. I guess the difference is that I virtualized
the dev1/2/3 form and used tooling to just let people work on master
directly, or let dev1/2/3 be concrete as a temporary form for
convenience, as I noted above.
Or in simpler terms, the model differences seem to boil down to this.
Given N possible targets:
1) Master is concrete, dev1..M (where M possibly > N) are generated
simply for ease of use either as concrete tree on disk or virtually in IDE
2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
Nation Status so something coherent can be shoved into SVN (Q: is MFNS
necessary? Nice, yes, but necessary?)
geir
Re: [classlib] Preprocessor - an attempt at clarity
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> Tim Ellison wrote:
>> There are multiple 'devtarget's -- each contains all the source code
>> marked-up for every target being developed, but each is distinct by
>> having different targets uncommented. Since each devtarget contains the
>> entire source code it is possible to accurately transform from any one
>> devtarget to any other devtarget. One of the devtargets is actually
>> stored in svn, and that is the canonical form of the devtarget code.
>
> So that's 'master'. and everythign else is a transformed master, right?
Yes, but some transforms preserve all mark-up and are used to switch the
compilable form between different targets (these are the 'devtarget's);
and some transforms are mark-up-lossy and are used to provide the end
user compilable form (this is the 'releasetarget').
>> Screech -- the releasetarget, by definition, does not contain the source
>> for all possible targets so that it is consumable by end-users.
>
> That is correct.
>
> The only code that contains all code is master.
Not in the technique that I'm talking about. All devtargets contain all
the code (but they differ in which code is compilable and which is in
comments).
Let me try using very simple example for my favourite type, assuming we
were doing CLDC and SE1.5 development simultaneously. The type
declaration for String is different for CLDC and SE1.5, and SE1.5 has
additional methods, etc.
If you are an SE developer you would compile and work in the devtarget
source for SE 1.5. Notice that we have a @process mark-up tag in both
type declaration comments, and since this is devtarget SE1.5 the CLDC
type declaration is commented out.
package java.lang;
/* A String is an immutable sequence of chars.
*
* @Process(CLDC)
* public final class String
*/
/* A String is an immutable, serializable, comparable
* sequence of chars.
*
* @Process(SE1.5)
*/
public final class String implements Serializable,
Comparable<String>, CharSequence
{
/*
* This constructor is common to both so needs no mark-up
*/
public String () {
value = new char[0];
offset = 0;
count = 0;
}
/*
* This method is SE1.5 only so needs to be marked.
* @Process(SE1.5)
*/
public String[] split(String expr) {
return Pattern.compile(expr).split(this);
}
}
Now if you want to develop CLDC you don't want to do that by editing
comments, you want the CLDC code to be uncommented so you can compile
it, but you want to preserve the SE1.5 code so you can get back to the
SE1.5 version accurately.
I probably don't need to show you, but for completeness here's the
result of transforming to the CLDC devstream. Note that the type
declaration comments flip-round, and the SE1.5-only method becomes
commented out.
package java.lang;
/* A String is an immutable sequence of chars.
*
* @Process(CLDC)
*/
public final class String
/* A String is an immutable, serializable, comparable
* sequence of chars.
*
* @Process(SE1.5)
* public final class String implements Serializable,
* Comparable<String>, CharSequence
*/
{
/*
* This constructor is common to both so needs no mark-up
*/
public String () {
value = new char[0];
offset = 0;
count = 0;
}
/*
* This method is SE1.5 only so needs to be marked.
* @Process(SE1.5)
*
* public String[] split(String expr) {
* return Pattern.compile(expr).split(this);
* }
*/
}
The big block comment for the split() method can be collapsed by the IDE
to a single line so you don't loose it but it is unobtrusive.
Now Mr CLDC developer can work in this stream and then transform back to
the canonical SE1.5 format and check-in his changes in. Storing in a
canonical format means we maintain a sane single HEAD stream in SVN.
Of course, Mrs CLDC user doesn't want to see all the SE1.5 comment
nonsense, so we can destructively transform any devsource into a
releasetarget CLDC for her. Again for completeness, here is the
releasetarget form of CLDC:
package java.lang;
/* A String is an immutable sequence of chars.
*/
public final class String
{
/*
* This constructor is common to both so needs no mark-up
*/
public String () {
value = new char[0];
offset = 0;
count = 0;
}
}
We then talked about how the different devtargets can be stored in
different working directories for use by Harmony developers with
file-based tools, or (thinking about it for a second now) it could be an
on the fly processing to/from canonical form on disk by the IDE editor.
We also talked about how patches to the source that Mrs CLDC developer
sees can be matched back to a devtarget.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
So this has been a long and exciting thread, and it's still not clear
that we understand each other. There are at least two different models
here.
I'll go re-read and try and clarify the difference, and I suppose the
next step for Etienne would be an example of how his approach works. It
will be fun to play with.
geir
Geir Magnusson Jr. wrote:
>
>
> Tim Ellison wrote:
>> Geir Magnusson Jr. wrote:
>>> Etienne Gagnon wrote:
>>>> Geir Magnusson Jr. wrote:
>>>>>> The "communication" aspect of 2 can be quite helpful when doing
>>>>>> system-wide changes. Just think about the effect of simply doing a
>>>>>> system-wide reindentation of source code; this is a nightmare when
>>>>>> developing using branches, as diff/merge tools are line-based, not
>>>>>> syntax-based. So, if your new indenter moves things across lines,
>>>>>> you're in hell.
>>>>>>
>>>>> So here's where I'm starting to believe I'm missing something
>>>>> important :
>>>>>
>>>>> Can you give me an example of the "communication" that you imagine
>>>>> taking place here? That's the part I'm just not grokking here.
>>>> How about: j2se5 developers, seeing that his planned modification to
>>>> some code will lead to utterly broken nearby j2se6 (commented) code
>>>> fragments, steps on this mailing list and asks j2se6 developers to help
>>>> him find the most harmless way to do the change (or, at least,warn them
>>>> about the problem)?
>>> Ok - we do that now. I thought you were saying that your tool added
>>> somehow to communications.
>>>
>>>>>> I am not saying that using the tool should replace using branches for
>>>>>> all situations, but there are situations where it would be more
>>>>>> effective not to use branches when the code is mostly identical.
>>>>> I want to avoid branches at all costs.
>>>> [somewhat off-topic] I think that we should consider tools/approaches
>>>> on their merit. Sometimes, branches are the right solution... You
>>>> wouldn't want to use syntax-processing instead of branches for managing
>>>> sandboxes. ;-)
>>> I'm trying to figure out the merit here.
>>
>> I believe that there will still be a role for branches, both for
>> sandboxes to experiment as Etienne said, and for the maintenance streams
>> of our releases.
>
> Of course we aren't going to ban branches. The context is branching for
> mainline code. Lets stay focused...
>
>>
>>>>> Clearly there's some confusion here. My goal is to
>>>>>
>>>>> 1) Avoid branches at all costs so we can share as much code, and
>>>>> get as
>>>>> much benefit for collaboration between different versions, and
>>>>> different
>>>>> platforms, if that happens.
>>>> Agreed.
>>>>
>>>>> 2) make it simple to work in either the 'master' code, or the 'target'
>>>>> code through tooling, including standard IDE activities like debugging
>>>> Agreed.
>>>>
>>>>> 3) make it easy for users to report bugs based on 'target' code,
>>>>> including patches and line numbers
>>>>
>>>> Ah! That's something to add to my requirements. Fine! I hadn't
>>>> included the "patches" thing into account. It doesn't break what I've
>>>> exposed so far; it simply adds to it.
>>> It has to be the case - we'll do snapshots and distributions of
>>> "src.jar" and when a developer goes into the debugger, they need to see
>>> normal Java SE 5 code.
>>
>> Agreed -- our src.jar will be 'normal' Java code without multi-target
>> mark-ups.
>>
>> <snip>
>>
>>>> That's neat. I like it. Yet, we would encourage developers to work
>>>> and
>>>> submit patches using devtarget code, instead of releasetarget code.
>>> I don't know this terminology. I was using "master" being the code in
>>> SVN, and "target", being the code Y, so the map is :
>>>
>>> master == devtarget
>>> target == releasetarget
>>>
>>> right? Ok.
>>
>> There are multiple 'devtarget's -- each contains all the source code
>> marked-up for every target being developed, but each is distinct by
>> having different targets uncommented. Since each devtarget contains the
>> entire source code it is possible to accurately transform from any one
>> devtarget to any other devtarget. One of the devtargets is actually
>> stored in svn, and that is the canonical form of the devtarget code.
>
> So that's 'master'. and everythign else is a transformed master, right?
>
>
>>
>>> I want to work in the master code w/ an IDE plugin that lets me think
>>> I'm in target (and lets me flip back to master). No preprocessing of
>>> the tree is required to develop.
>>>
>>> However, end-users - people who take our JDK and work with it, will
>>> report bugs with stacktraces and line numbers that are from
>>> target/releasetarget code.
>>
>> Screech -- the releasetarget, by definition, does not contain the source
>> for all possible targets so that it is consumable by end-users.
>
> That is correct.
>
> The only code that contains all code is master.
>
>
>>
>>> So what to do with a patch?
>>>
>>> I can either
>>>
>>> patch -p0 << reverse(patch.file)
>>
>> I don't understand how you will ensure there is enough information in
>> patch.file to make the reverse() function exact? I proposed that our
>> releasetarget code contains tie-points where it is out of sync with the
>> closest devtarget by some number of lines.
>
> I don't understand why you'd bother. If you know that patch P creates
> Y' from a given Y, and you can always create Y as you know everything to
> do the xform from master source and the platform name, everything is known.
>
>>
>>> on the main code or use patching facilities in an IDE to patch the
>>> transform view
>>>
>>> Man, this tooling is going to be fancy! :/
>>>
>>>
>>>> In other words, here's how I see the distribution(s):
>>>>
>>>> - Binary j2se5 Harmony release: includes j2se5release src.jar for
>>>> end-developers using debuggers to step through API code.
>>>>
>>>> - Source j2se5 Harmony release: API are in j2se5dev form. The build
>>>> process uses the processing tool to generate j2se5release src.jar.
>>>>
>>>> - Binary j2se6 Harmony release: includes j2se6release src.jar for
>>>> end-developers using debuggers to step through API code.
>>>>
>>>> - Source j2se5 Harmony release: API are in j2se6dev form. The build
>>>> process uses the processing tool to generate j2se6release src.jar.
>>> Yes.
>>>
>>>> Why? Because reverse works much better with devtarget than
>>>> releasetarget code, and the communications benefit of devtarget that
>>>> are
>>>> lost in releasetarget code (because of stream code erasure). In
>>>> addition, using smart IDEs, there wouldn't be much difference for
>>>> developers between the "visible" formats of dev vs release code when
>>>> working on it.
>>> I don't think it matters, as I think that most people interested in
>>> working on the code will just check out of SVN.
>>>
>>>>> for X being a single class. To produce a complete target source base,
>>>>> walk the single source tree as defined by a 'platform descriptor' :
>>>> I don't really understand the "platform descriptor" thing, yet I think
>>>> that it is, somehow, an orthogonal issue to the discussion above.
>>>> [Remember my long message describing 2 levels of processing:
>>>> file/directory-level and source-level.] My first prototype tool would
>>>> attack source-level processing (discussed above), to validate the
>>>> approach.
>>> Right - the "platform descriptor" is the data that the
>>> file/directory-level thing uses.
>>>
>>>> Maybe you could try rephrasing your file/directory-level (right?)
>>>> proposal? How does it differ from Tim's proposal?
>>> I get lost in who's proposal is who's. I thought we were working
>>> towards one collectively owned solution. can you describe what you
>>> think Tim's, your's and mine are?
>>
>> It's all one proposal. Etienne's principal idea with a few others
>> meddling on the side :-)
>>
>> Regards,
>> Tim
>>
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Etienne Gagnon wrote:
>>> Geir Magnusson Jr. wrote:
>>>>> The "communication" aspect of 2 can be quite helpful when doing
>>>>> system-wide changes. Just think about the effect of simply doing a
>>>>> system-wide reindentation of source code; this is a nightmare when
>>>>> developing using branches, as diff/merge tools are line-based, not
>>>>> syntax-based. So, if your new indenter moves things across lines,
>>>>> you're in hell.
>>>>>
>>>> So here's where I'm starting to believe I'm missing something
>>>> important :
>>>>
>>>> Can you give me an example of the "communication" that you imagine
>>>> taking place here? That's the part I'm just not grokking here.
>>> How about: j2se5 developers, seeing that his planned modification to
>>> some code will lead to utterly broken nearby j2se6 (commented) code
>>> fragments, steps on this mailing list and asks j2se6 developers to help
>>> him find the most harmless way to do the change (or, at least,warn them
>>> about the problem)?
>> Ok - we do that now. I thought you were saying that your tool added
>> somehow to communications.
>>
>>>>> I am not saying that using the tool should replace using branches for
>>>>> all situations, but there are situations where it would be more
>>>>> effective not to use branches when the code is mostly identical.
>>>> I want to avoid branches at all costs.
>>> [somewhat off-topic] I think that we should consider tools/approaches
>>> on their merit. Sometimes, branches are the right solution... You
>>> wouldn't want to use syntax-processing instead of branches for managing
>>> sandboxes. ;-)
>> I'm trying to figure out the merit here.
>
> I believe that there will still be a role for branches, both for
> sandboxes to experiment as Etienne said, and for the maintenance streams
> of our releases.
Of course we aren't going to ban branches. The context is branching for
mainline code. Lets stay focused...
>
>>>> Clearly there's some confusion here. My goal is to
>>>>
>>>> 1) Avoid branches at all costs so we can share as much code, and get as
>>>> much benefit for collaboration between different versions, and different
>>>> platforms, if that happens.
>>> Agreed.
>>>
>>>> 2) make it simple to work in either the 'master' code, or the 'target'
>>>> code through tooling, including standard IDE activities like debugging
>>> Agreed.
>>>
>>>> 3) make it easy for users to report bugs based on 'target' code,
>>>> including patches and line numbers
>>>
>>> Ah! That's something to add to my requirements. Fine! I hadn't
>>> included the "patches" thing into account. It doesn't break what I've
>>> exposed so far; it simply adds to it.
>> It has to be the case - we'll do snapshots and distributions of
>> "src.jar" and when a developer goes into the debugger, they need to see
>> normal Java SE 5 code.
>
> Agreed -- our src.jar will be 'normal' Java code without multi-target
> mark-ups.
>
> <snip>
>
>>> That's neat. I like it. Yet, we would encourage developers to work and
>>> submit patches using devtarget code, instead of releasetarget code.
>> I don't know this terminology. I was using "master" being the code in
>> SVN, and "target", being the code Y, so the map is :
>>
>> master == devtarget
>> target == releasetarget
>>
>> right? Ok.
>
> There are multiple 'devtarget's -- each contains all the source code
> marked-up for every target being developed, but each is distinct by
> having different targets uncommented. Since each devtarget contains the
> entire source code it is possible to accurately transform from any one
> devtarget to any other devtarget. One of the devtargets is actually
> stored in svn, and that is the canonical form of the devtarget code.
So that's 'master'. and everythign else is a transformed master, right?
>
>> I want to work in the master code w/ an IDE plugin that lets me think
>> I'm in target (and lets me flip back to master). No preprocessing of
>> the tree is required to develop.
>>
>> However, end-users - people who take our JDK and work with it, will
>> report bugs with stacktraces and line numbers that are from
>> target/releasetarget code.
>
> Screech -- the releasetarget, by definition, does not contain the source
> for all possible targets so that it is consumable by end-users.
That is correct.
The only code that contains all code is master.
>
>> So what to do with a patch?
>>
>> I can either
>>
>> patch -p0 << reverse(patch.file)
>
> I don't understand how you will ensure there is enough information in
> patch.file to make the reverse() function exact? I proposed that our
> releasetarget code contains tie-points where it is out of sync with the
> closest devtarget by some number of lines.
I don't understand why you'd bother. If you know that patch P creates
Y' from a given Y, and you can always create Y as you know everything to
do the xform from master source and the platform name, everything is known.
>
>> on the main code or use patching facilities in an IDE to patch the
>> transform view
>>
>> Man, this tooling is going to be fancy! :/
>>
>>
>>> In other words, here's how I see the distribution(s):
>>>
>>> - Binary j2se5 Harmony release: includes j2se5release src.jar for
>>> end-developers using debuggers to step through API code.
>>>
>>> - Source j2se5 Harmony release: API are in j2se5dev form. The build
>>> process uses the processing tool to generate j2se5release src.jar.
>>>
>>> - Binary j2se6 Harmony release: includes j2se6release src.jar for
>>> end-developers using debuggers to step through API code.
>>>
>>> - Source j2se5 Harmony release: API are in j2se6dev form. The build
>>> process uses the processing tool to generate j2se6release src.jar.
>> Yes.
>>
>>> Why? Because reverse works much better with devtarget than
>>> releasetarget code, and the communications benefit of devtarget that are
>>> lost in releasetarget code (because of stream code erasure). In
>>> addition, using smart IDEs, there wouldn't be much difference for
>>> developers between the "visible" formats of dev vs release code when
>>> working on it.
>> I don't think it matters, as I think that most people interested in
>> working on the code will just check out of SVN.
>>
>>>> for X being a single class. To produce a complete target source base,
>>>> walk the single source tree as defined by a 'platform descriptor' :
>>> I don't really understand the "platform descriptor" thing, yet I think
>>> that it is, somehow, an orthogonal issue to the discussion above.
>>> [Remember my long message describing 2 levels of processing:
>>> file/directory-level and source-level.] My first prototype tool would
>>> attack source-level processing (discussed above), to validate the
>>> approach.
>> Right - the "platform descriptor" is the data that the
>> file/directory-level thing uses.
>>
>>> Maybe you could try rephrasing your file/directory-level (right?)
>>> proposal? How does it differ from Tim's proposal?
>> I get lost in who's proposal is who's. I thought we were working
>> towards one collectively owned solution. can you describe what you
>> think Tim's, your's and mine are?
>
> It's all one proposal. Etienne's principal idea with a few others
> meddling on the side :-)
>
> Regards,
> Tim
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> Etienne Gagnon wrote:
>> Geir Magnusson Jr. wrote:
>>>> The "communication" aspect of 2 can be quite helpful when doing
>>>> system-wide changes. Just think about the effect of simply doing a
>>>> system-wide reindentation of source code; this is a nightmare when
>>>> developing using branches, as diff/merge tools are line-based, not
>>>> syntax-based. So, if your new indenter moves things across lines,
>>>> you're in hell.
>>>>
>>> So here's where I'm starting to believe I'm missing something
>>> important :
>>>
>>> Can you give me an example of the "communication" that you imagine
>>> taking place here? That's the part I'm just not grokking here.
>>
>> How about: j2se5 developers, seeing that his planned modification to
>> some code will lead to utterly broken nearby j2se6 (commented) code
>> fragments, steps on this mailing list and asks j2se6 developers to help
>> him find the most harmless way to do the change (or, at least,warn them
>> about the problem)?
>
> Ok - we do that now. I thought you were saying that your tool added
> somehow to communications.
>
>>
>>>> I am not saying that using the tool should replace using branches for
>>>> all situations, but there are situations where it would be more
>>>> effective not to use branches when the code is mostly identical.
>>> I want to avoid branches at all costs.
>>
>> [somewhat off-topic] I think that we should consider tools/approaches
>> on their merit. Sometimes, branches are the right solution... You
>> wouldn't want to use syntax-processing instead of branches for managing
>> sandboxes. ;-)
>
> I'm trying to figure out the merit here.
I believe that there will still be a role for branches, both for
sandboxes to experiment as Etienne said, and for the maintenance streams
of our releases.
>>> Clearly there's some confusion here. My goal is to
>>>
>>> 1) Avoid branches at all costs so we can share as much code, and get as
>>> much benefit for collaboration between different versions, and different
>>> platforms, if that happens.
>>
>> Agreed.
>>
>>> 2) make it simple to work in either the 'master' code, or the 'target'
>>> code through tooling, including standard IDE activities like debugging
>>
>> Agreed.
>>
>>> 3) make it easy for users to report bugs based on 'target' code,
>>> including patches and line numbers
>>
>>
>> Ah! That's something to add to my requirements. Fine! I hadn't
>> included the "patches" thing into account. It doesn't break what I've
>> exposed so far; it simply adds to it.
>
> It has to be the case - we'll do snapshots and distributions of
> "src.jar" and when a developer goes into the debugger, they need to see
> normal Java SE 5 code.
Agreed -- our src.jar will be 'normal' Java code without multi-target
mark-ups.
<snip>
>> That's neat. I like it. Yet, we would encourage developers to work and
>> submit patches using devtarget code, instead of releasetarget code.
>
> I don't know this terminology. I was using "master" being the code in
> SVN, and "target", being the code Y, so the map is :
>
> master == devtarget
> target == releasetarget
>
> right? Ok.
There are multiple 'devtarget's -- each contains all the source code
marked-up for every target being developed, but each is distinct by
having different targets uncommented. Since each devtarget contains the
entire source code it is possible to accurately transform from any one
devtarget to any other devtarget. One of the devtargets is actually
stored in svn, and that is the canonical form of the devtarget code.
> I want to work in the master code w/ an IDE plugin that lets me think
> I'm in target (and lets me flip back to master). No preprocessing of
> the tree is required to develop.
>
> However, end-users - people who take our JDK and work with it, will
> report bugs with stacktraces and line numbers that are from
> target/releasetarget code.
Screech -- the releasetarget, by definition, does not contain the source
for all possible targets so that it is consumable by end-users.
> So what to do with a patch?
>
> I can either
>
> patch -p0 << reverse(patch.file)
I don't understand how you will ensure there is enough information in
patch.file to make the reverse() function exact? I proposed that our
releasetarget code contains tie-points where it is out of sync with the
closest devtarget by some number of lines.
> on the main code or use patching facilities in an IDE to patch the
> transform view
>
> Man, this tooling is going to be fancy! :/
>
>
>>
>> In other words, here's how I see the distribution(s):
>>
>> - Binary j2se5 Harmony release: includes j2se5release src.jar for
>> end-developers using debuggers to step through API code.
>>
>> - Source j2se5 Harmony release: API are in j2se5dev form. The build
>> process uses the processing tool to generate j2se5release src.jar.
>>
>> - Binary j2se6 Harmony release: includes j2se6release src.jar for
>> end-developers using debuggers to step through API code.
>>
>> - Source j2se5 Harmony release: API are in j2se6dev form. The build
>> process uses the processing tool to generate j2se6release src.jar.
>
> Yes.
>
>>
>> Why? Because reverse works much better with devtarget than
>> releasetarget code, and the communications benefit of devtarget that are
>> lost in releasetarget code (because of stream code erasure). In
>> addition, using smart IDEs, there wouldn't be much difference for
>> developers between the "visible" formats of dev vs release code when
>> working on it.
>
> I don't think it matters, as I think that most people interested in
> working on the code will just check out of SVN.
>
>>
>>> for X being a single class. To produce a complete target source base,
>>> walk the single source tree as defined by a 'platform descriptor' :
>>
>> I don't really understand the "platform descriptor" thing, yet I think
>> that it is, somehow, an orthogonal issue to the discussion above.
>> [Remember my long message describing 2 levels of processing:
>> file/directory-level and source-level.] My first prototype tool would
>> attack source-level processing (discussed above), to validate the
>> approach.
>
> Right - the "platform descriptor" is the data that the
> file/directory-level thing uses.
>
>>
>> Maybe you could try rephrasing your file/directory-level (right?)
>> proposal? How does it differ from Tim's proposal?
>
> I get lost in who's proposal is who's. I thought we were working
> towards one collectively owned solution. can you describe what you
> think Tim's, your's and mine are?
It's all one proposal. Etienne's principal idea with a few others
meddling on the side :-)
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
BTW, I'm more than happy if you just do a quick sketch of what you're
thinking, and we resume the convo from there.
I just wanted to clear up some confusion I had.
geir
Geir Magnusson Jr. wrote:
>
>
> Etienne Gagnon wrote:
>> Geir Magnusson Jr. wrote:
>>>> The "communication" aspect of 2 can be quite helpful when doing
>>>> system-wide changes. Just think about the effect of simply doing a
>>>> system-wide reindentation of source code; this is a nightmare when
>>>> developing using branches, as diff/merge tools are line-based, not
>>>> syntax-based. So, if your new indenter moves things across lines,
>>>> you're in hell.
>>>>
>>> So here's where I'm starting to believe I'm missing something
>>> important :
>>>
>>> Can you give me an example of the "communication" that you imagine
>>> taking place here? That's the part I'm just not grokking here.
>>
>> How about: j2se5 developers, seeing that his planned modification to
>> some code will lead to utterly broken nearby j2se6 (commented) code
>> fragments, steps on this mailing list and asks j2se6 developers to help
>> him find the most harmless way to do the change (or, at least,warn them
>> about the problem)?
>
> Ok - we do that now. I thought you were saying that your tool added
> somehow to communications.
>
>>
>>>> I am not saying that using the tool should replace using branches for
>>>> all situations, but there are situations where it would be more
>>>> effective not to use branches when the code is mostly identical.
>>> I want to avoid branches at all costs.
>>
>> [somewhat off-topic] I think that we should consider tools/approaches
>> on their merit. Sometimes, branches are the right solution... You
>> wouldn't want to use syntax-processing instead of branches for managing
>> sandboxes. ;-)
>
> I'm trying to figure out the merit here.
>
>>
>>> Clearly there's some confusion here. My goal is to
>>>
>>> 1) Avoid branches at all costs so we can share as much code, and get as
>>> much benefit for collaboration between different versions, and different
>>> platforms, if that happens.
>>
>> Agreed.
>>
>>> 2) make it simple to work in either the 'master' code, or the 'target'
>>> code through tooling, including standard IDE activities like debugging
>>
>> Agreed.
>>
>>> 3) make it easy for users to report bugs based on 'target' code,
>>> including patches and line numbers
>>
>>
>> Ah! That's something to add to my requirements. Fine! I hadn't
>> included the "patches" thing into account. It doesn't break what I've
>> exposed so far; it simply adds to it.
>
> It has to be the case - we'll do snapshots and distributions of
> "src.jar" and when a developer goes into the debugger, they need to see
> normal Java SE 5 code.
>
>>
>>> What I've suggested is :
>>>
>>> forward(X, platform) -> Y
>>>
>>> reverse(X, platform, Y') -> X'
>>
>> OK. I see this in *addition* to:
>>
>> forward(X, devtarget) -> Y
>> reverse(Y') -> X'
>
> I believe that
>
> reverse(X, palatform, Y') -> X'
>
> is the same as
>
> reverse(Y') -> X'
>
> as you need to know at least what platform Y' is, and what it came from.
> You could always embed as metadata in the code itself in a comment, I
> suppose. I was just being explicit.
>
>
>>
>> So, to simplify the discussion, let's rewrite your proposal as:
>>
>> forward(X, releasetarget) -> Y
>> reverse(Y',X,releasetarget) ~~> X' (possibly reporting
>> conflicts/problems)
>>
>> That's neat. I like it. Yet, we would encourage developers to work and
>> submit patches using devtarget code, instead of releasetarget code.
>
> I don't know this terminology. I was using "master" being the code in
> SVN, and "target", being the code Y, so the map is :
>
> master == devtarget
> target == releasetarget
>
> right? Ok.
>
> I want to work in the master code w/ an IDE plugin that lets me think
> I'm in target (and lets me flip back to master). No preprocessing of
> the tree is required to develop.
>
> However, end-users - people who take our JDK and work with it, will
> report bugs with stacktraces and line numbers that are from
> target/releasetarget code.
>
> So what to do with a patch?
>
> I can either
>
> patch -p0 << reverse(patch.file)
>
> on the main code or use patching facilities in an IDE to patch the
> transform view
>
> Man, this tooling is going to be fancy! :/
>
>
>>
>> In other words, here's how I see the distribution(s):
>>
>> - Binary j2se5 Harmony release: includes j2se5release src.jar for
>> end-developers using debuggers to step through API code.
>>
>> - Source j2se5 Harmony release: API are in j2se5dev form. The build
>> process uses the processing tool to generate j2se5release src.jar.
>>
>> - Binary j2se6 Harmony release: includes j2se6release src.jar for
>> end-developers using debuggers to step through API code.
>>
>> - Source j2se5 Harmony release: API are in j2se6dev form. The build
>> process uses the processing tool to generate j2se6release src.jar.
>
> Yes.
>
>>
>> Why? Because reverse works much better with devtarget than
>> releasetarget code, and the communications benefit of devtarget that are
>> lost in releasetarget code (because of stream code erasure). In
>> addition, using smart IDEs, there wouldn't be much difference for
>> developers between the "visible" formats of dev vs release code when
>> working on it.
>
> I don't think it matters, as I think that most people interested in
> working on the code will just check out of SVN.
>
>>
>>> for X being a single class. To produce a complete target source base,
>>> walk the single source tree as defined by a 'platform descriptor' :
>>
>> I don't really understand the "platform descriptor" thing, yet I think
>> that it is, somehow, an orthogonal issue to the discussion above.
>> [Remember my long message describing 2 levels of processing:
>> file/directory-level and source-level.] My first prototype tool would
>> attack source-level processing (discussed above), to validate the
>> approach.
>
> Right - the "platform descriptor" is the data that the
> file/directory-level thing uses.
>
>>
>> Maybe you could try rephrasing your file/directory-level (right?)
>> proposal? How does it differ from Tim's proposal?
>
> I get lost in who's proposal is who's. I thought we were working
> towards one collectively owned solution. can you describe what you
> think Tim's, your's and mine are?
>
> geir
>
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>>> The "communication" aspect of 2 can be quite helpful when doing
>>> system-wide changes. Just think about the effect of simply doing a
>>> system-wide reindentation of source code; this is a nightmare when
>>> developing using branches, as diff/merge tools are line-based, not
>>> syntax-based. So, if your new indenter moves things across lines,
>>> you're in hell.
>>>
>> So here's where I'm starting to believe I'm missing something important :
>>
>> Can you give me an example of the "communication" that you imagine
>> taking place here? That's the part I'm just not grokking here.
>
> How about: j2se5 developers, seeing that his planned modification to
> some code will lead to utterly broken nearby j2se6 (commented) code
> fragments, steps on this mailing list and asks j2se6 developers to help
> him find the most harmless way to do the change (or, at least,warn them
> about the problem)?
Ok - we do that now. I thought you were saying that your tool added
somehow to communications.
>
>>> I am not saying that using the tool should replace using branches for
>>> all situations, but there are situations where it would be more
>>> effective not to use branches when the code is mostly identical.
>> I want to avoid branches at all costs.
>
> [somewhat off-topic] I think that we should consider tools/approaches
> on their merit. Sometimes, branches are the right solution... You
> wouldn't want to use syntax-processing instead of branches for managing
> sandboxes. ;-)
I'm trying to figure out the merit here.
>
>> Clearly there's some confusion here. My goal is to
>>
>> 1) Avoid branches at all costs so we can share as much code, and get as
>> much benefit for collaboration between different versions, and different
>> platforms, if that happens.
>
> Agreed.
>
>> 2) make it simple to work in either the 'master' code, or the 'target'
>> code through tooling, including standard IDE activities like debugging
>
> Agreed.
>
>> 3) make it easy for users to report bugs based on 'target' code,
>> including patches and line numbers
>
>
> Ah! That's something to add to my requirements. Fine! I hadn't
> included the "patches" thing into account. It doesn't break what I've
> exposed so far; it simply adds to it.
It has to be the case - we'll do snapshots and distributions of
"src.jar" and when a developer goes into the debugger, they need to see
normal Java SE 5 code.
>
>> What I've suggested is :
>>
>> forward(X, platform) -> Y
>>
>> reverse(X, platform, Y') -> X'
>
> OK. I see this in *addition* to:
>
> forward(X, devtarget) -> Y
> reverse(Y') -> X'
I believe that
reverse(X, palatform, Y') -> X'
is the same as
reverse(Y') -> X'
as you need to know at least what platform Y' is, and what it came from.
You could always embed as metadata in the code itself in a comment, I
suppose. I was just being explicit.
>
> So, to simplify the discussion, let's rewrite your proposal as:
>
> forward(X, releasetarget) -> Y
> reverse(Y',X,releasetarget) ~~> X' (possibly reporting
> conflicts/problems)
>
> That's neat. I like it. Yet, we would encourage developers to work and
> submit patches using devtarget code, instead of releasetarget code.
I don't know this terminology. I was using "master" being the code in
SVN, and "target", being the code Y, so the map is :
master == devtarget
target == releasetarget
right? Ok.
I want to work in the master code w/ an IDE plugin that lets me think
I'm in target (and lets me flip back to master). No preprocessing of
the tree is required to develop.
However, end-users - people who take our JDK and work with it, will
report bugs with stacktraces and line numbers that are from
target/releasetarget code.
So what to do with a patch?
I can either
patch -p0 << reverse(patch.file)
on the main code or use patching facilities in an IDE to patch the
transform view
Man, this tooling is going to be fancy! :/
>
> In other words, here's how I see the distribution(s):
>
> - Binary j2se5 Harmony release: includes j2se5release src.jar for
> end-developers using debuggers to step through API code.
>
> - Source j2se5 Harmony release: API are in j2se5dev form. The build
> process uses the processing tool to generate j2se5release src.jar.
>
> - Binary j2se6 Harmony release: includes j2se6release src.jar for
> end-developers using debuggers to step through API code.
>
> - Source j2se5 Harmony release: API are in j2se6dev form. The build
> process uses the processing tool to generate j2se6release src.jar.
Yes.
>
> Why? Because reverse works much better with devtarget than
> releasetarget code, and the communications benefit of devtarget that are
> lost in releasetarget code (because of stream code erasure). In
> addition, using smart IDEs, there wouldn't be much difference for
> developers between the "visible" formats of dev vs release code when
> working on it.
I don't think it matters, as I think that most people interested in
working on the code will just check out of SVN.
>
>> for X being a single class. To produce a complete target source base,
>> walk the single source tree as defined by a 'platform descriptor' :
>
> I don't really understand the "platform descriptor" thing, yet I think
> that it is, somehow, an orthogonal issue to the discussion above.
> [Remember my long message describing 2 levels of processing:
> file/directory-level and source-level.] My first prototype tool would
> attack source-level processing (discussed above), to validate the approach.
Right - the "platform descriptor" is the data that the
file/directory-level thing uses.
>
> Maybe you could try rephrasing your file/directory-level (right?)
> proposal? How does it differ from Tim's proposal?
I get lost in who's proposal is who's. I thought we were working
towards one collectively owned solution. can you describe what you
think Tim's, your's and mine are?
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
>> The "communication" aspect of 2 can be quite helpful when doing
>> system-wide changes. Just think about the effect of simply doing a
>> system-wide reindentation of source code; this is a nightmare when
>> developing using branches, as diff/merge tools are line-based, not
>> syntax-based. So, if your new indenter moves things across lines,
>> you're in hell.
>>
>
> So here's where I'm starting to believe I'm missing something important :
>
> Can you give me an example of the "communication" that you imagine
> taking place here? That's the part I'm just not grokking here.
How about: j2se5 developers, seeing that his planned modification to
some code will lead to utterly broken nearby j2se6 (commented) code
fragments, steps on this mailing list and asks j2se6 developers to help
him find the most harmless way to do the change (or, at least,warn them
about the problem)?
>> I am not saying that using the tool should replace using branches for
>> all situations, but there are situations where it would be more
>> effective not to use branches when the code is mostly identical.
>
> I want to avoid branches at all costs.
[somewhat off-topic] I think that we should consider tools/approaches
on their merit. Sometimes, branches are the right solution... You
wouldn't want to use syntax-processing instead of branches for managing
sandboxes. ;-)
> Clearly there's some confusion here. My goal is to
>
> 1) Avoid branches at all costs so we can share as much code, and get as
> much benefit for collaboration between different versions, and different
> platforms, if that happens.
Agreed.
> 2) make it simple to work in either the 'master' code, or the 'target'
> code through tooling, including standard IDE activities like debugging
Agreed.
>
> 3) make it easy for users to report bugs based on 'target' code,
> including patches and line numbers
Ah! That's something to add to my requirements. Fine! I hadn't
included the "patches" thing into account. It doesn't break what I've
exposed so far; it simply adds to it.
> What I've suggested is :
>
> forward(X, platform) -> Y
>
> reverse(X, platform, Y') -> X'
OK. I see this in *addition* to:
forward(X, devtarget) -> Y
reverse(Y') -> X'
So, to simplify the discussion, let's rewrite your proposal as:
forward(X, releasetarget) -> Y
reverse(Y',X,releasetarget) ~~> X' (possibly reporting
conflicts/problems)
That's neat. I like it. Yet, we would encourage developers to work and
submit patches using devtarget code, instead of releasetarget code.
In other words, here's how I see the distribution(s):
- Binary j2se5 Harmony release: includes j2se5release src.jar for
end-developers using debuggers to step through API code.
- Source j2se5 Harmony release: API are in j2se5dev form. The build
process uses the processing tool to generate j2se5release src.jar.
- Binary j2se6 Harmony release: includes j2se6release src.jar for
end-developers using debuggers to step through API code.
- Source j2se5 Harmony release: API are in j2se6dev form. The build
process uses the processing tool to generate j2se6release src.jar.
Why? Because reverse works much better with devtarget than
releasetarget code, and the communications benefit of devtarget that are
lost in releasetarget code (because of stream code erasure). In
addition, using smart IDEs, there wouldn't be much difference for
developers between the "visible" formats of dev vs release code when
working on it.
> for X being a single class. To produce a complete target source base,
> walk the single source tree as defined by a 'platform descriptor' :
I don't really understand the "platform descriptor" thing, yet I think
that it is, somehow, an orthogonal issue to the discussion above.
[Remember my long message describing 2 levels of processing:
file/directory-level and source-level.] My first prototype tool would
attack source-level processing (discussed above), to validate the approach.
Maybe you could try rephrasing your file/directory-level (right?)
proposal? How does it differ from Tim's proposal?
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
[SNIP]
>
> Such changes are usually not dealt with very neatly with "svn merge".
I'm confused. I don't think anyone has suggested that we use "svn
merge" for this.
> Now, if using the processign tool, you get at least 2 benefits over "svn
> merge":
> 1- No need to do any "svn merge"... The method was moved AND renamed
> even for j2se6.
> 2- You were aware, when moving the method and changing its name that you
> could possibly be making the life of j2se6 developers a little more
> difficult.
>
Right - that's how I thought it would work.
> The "communication" aspect of 2 can be quite helpful when doing
> system-wide changes. Just think about the effect of simply doing a
> system-wide reindentation of source code; this is a nightmare when
> developing using branches, as diff/merge tools are line-based, not
> syntax-based. So, if your new indenter moves things across lines,
> you're in hell.
>
So here's where I'm starting to believe I'm missing something important :
Can you give me an example of the "communication" that you imagine
taking place here? That's the part I'm just not grokking here.
> Believe me; I've lived through that trying to maintain a "well indented"
> GNU Classpath in SableVM's repository.
>
> I am not saying that using the tool should replace using branches for
> all situations, but there are situations where it would be more
> effective not to use branches when the code is mostly identical.
I want to avoid branches at all costs.
> Using
> traditional pre-processing tools, in such cases, is often the solution
> adopted by some people; the problems of "simply-mided" traditiona
> pre-processing has been discussed earlier in this thread. I am
> proposing, instead, to use an innovative "syntax-based, revertable
> processing". That's all. [I guess you'll need to see it working to
> make sense of it... I'll work on a prototype. I've just submitted the
> ICLA/ACQ].
Syntax based is fine - we never settled on what kind of pre-processing
scheme we would use, so whatever seems best, go for it.
Clearly there's some confusion here. My goal is to
1) Avoid branches at all costs so we can share as much code, and get as
much benefit for collaboration between different versions, and different
platforms, if that happens.
2) make it simple to work in either the 'master' code, or the 'target'
code through tooling, including standard IDE activities like debugging
3) make it easy for users to report bugs based on 'target' code,
including patches and line numbers
Now, maybe I'm misunderstanding you, but it sounds like your approach is
one-way, and coarse grained.
What I've suggested is :
forward(X, platform) -> Y
reverse(X, platform, Y') -> X'
for X being a single class. To produce a complete target source base,
walk the single source tree as defined by a 'platform descriptor' :
foreach(file in descriptor) {
forward(X, platform);
}
so for a source tree
master/
dir1/
X1
X2
dir2/
X3
X4
and a platform descriptor
java5/
dir1/
X1
dir2/
X4
then walking the tree gets us :
java5/
dir1/
Y1
dir2/
Y4
Does this match your model? If not, what's different? I admit the
above is simple minded. I can imagine many, many messy corner cases.
But it's a reasonable starting point? I'd like to setup this 'master'
tree in the sandbox. Can anyone suggest any good examples of diffs
between java 5 and java 6 so we can put diffs in the classes X1-4, or
have paltform unique classes in X1-4?
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
>> I have nothing against this! But, we have to make sure:
>> 1- we don't lose the "communication" aspect of telling developers about
>> parrallel development.
>
> I'm not sure I understand what problem you are trying to solve there. We
> do parallel development on the source base right now - no one ever does
> a blocking checkout - it's optimistic.
What tells you, when you modify method m() of class C that,in the middle
of it, there's different ways of processing something in j2se6?
e.g.
class C
{
.../* many, many methods */
private void m()
{
... /* canonical j2se5 code */
...
// @Process(j2se6)
// { do something additional }
...
...
}
}
So, if you were, for example, to move m() to become the first method,
and ALSO change its name...
class C
{
private void m2()
{
... /* canonical j2se5 code with modifications!! */
...
// @Process(j2se6)
// { do something additional }
...
...
}
.../* many, many methods */
}
Such changes are usually not dealt with very neatly with "svn merge".
Now, if using the processign tool, you get at least 2 benefits over "svn
merge":
1- No need to do any "svn merge"... The method was moved AND renamed
even for j2se6.
2- You were aware, when moving the method and changing its name that you
could possibly be making the life of j2se6 developers a little more
difficult.
The "communication" aspect of 2 can be quite helpful when doing
system-wide changes. Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based. So, if your new indenter moves things across lines,
you're in hell.
Believe me; I've lived through that trying to maintain a "well indented"
GNU Classpath in SableVM's repository.
I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical. Using
traditional pre-processing tools, in such cases, is often the solution
adopted by some people; the problems of "simply-mided" traditiona
pre-processing has been discussed earlier in this thread. I am
proposing, instead, to use an innovative "syntax-based, revertable
processing". That's all. [I guess you'll need to see it working to
make sense of it... I'll work on a prototype. I've just submitted the
ICLA/ACQ].
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> [...]
>>
>> See, I'm hoping for something a tad different :
>>
>> 1) For building : process() (and revert() for fun) for cmd line use for
>> the build scripts, so we just do
>> ...
>>
>> 2) For development : IDE plugin where
>>
>> a) I can tell plugin that my project def/configuration is whatever,
>> it using metadata in a file, only consider the code that is
>> defined (or not excluded - whatever is easier)
>>
>> b) I can tell plugin to look at X, know that I want
>> "java5" and in situ, it shows me what Z would look like.
>> I edit this, but I'm really editing X. And I want to
>> be sure, I push a button or have a split screen that
>> shows me what X really looks like.
>>
>> I can edit in X, or in "virtual Z", or both at the same
>> time, as I'm just really editing X.
>
> I have nothing against this! But, we have to make sure:
> 1- we don't lose the "communication" aspect of telling developers about
> parrallel development.
I'm not sure I understand what problem you are trying to solve there.
We do parallel development on the source base right now - no one ever
does a blocking checkout - it's optimistic.
> 2- we have a robust "language" design; this is best achieved by first
> building the command-line based tool, then extend it into a full-blown
> Eclipse plugin... :-)
Right. The fact it's the same basic transformation code and process.
The fact that someone embedded it in eclipse doesn't change the
fundamental process.
geir
>
> Etienne
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
>[...]
>
> See, I'm hoping for something a tad different :
>
> 1) For building : process() (and revert() for fun) for cmd line use for
> the build scripts, so we just do
>...
>
> 2) For development : IDE plugin where
>
> a) I can tell plugin that my project def/configuration is whatever,
> it using metadata in a file, only consider the code that is
> defined (or not excluded - whatever is easier)
>
> b) I can tell plugin to look at X, know that I want
> "java5" and in situ, it shows me what Z would look like.
> I edit this, but I'm really editing X. And I want to
> be sure, I push a button or have a split screen that
> shows me what X really looks like.
>
> I can edit in X, or in "virtual Z", or both at the same
> time, as I'm just really editing X.
I have nothing against this! But, we have to make sure:
1- we don't lose the "communication" aspect of telling developers about
parrallel development.
2- we have a robust "language" design; this is best achieved by first
building the command-line based tool, then extend it into a full-blown
Eclipse plugin... :-)
Etienne
--
Etienne M. Gagnon, Ph.D. egagnon@j-meg.com
President http://www.sablevm.org/
J-Meg inc. http://www.sablecc.org/
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Mikhail Fursov wrote:
>> On 11/1/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>>> For comfortable IDE development, one could imagine that the IDE editor
>>> can reduce to "one-line visible" comments (or better, specially
>>> formatted ones) so that it gives you the impression that you are really
>>> wearing target-specific spectacles. [I know Eclipse allows for such
>>> things already].
>>> ...
>> Etienne,
>> What is 'comfortable IDE development' if you can't modify the Y? Am I
>> missing something here?
>
> Maybe my text was wrongly formatted... You looked at the wrong example.
> Comfortable development happens only using "development targets". E.g.
>
> 1- process(X, devtarget) -> Z
>
> 2- edit Z in IDE using comfortable development, where you see a single
> commented line for every hidden stream code chunk, keeping you aware
> that other streams have related code there [you click on the "+" in
> Eclipse if you want to see the complete chunk]. Of course, you
> should never delete a chunk without consulting other stream
> developers first.
> So: edit Z -> Z'
See, I'm hoping for something a tad different :
1) For building : process() (and revert() for fun) for cmd line use for
the build scripts, so we just do
$ ant -Dtarget=java5
and what pops out is the java5 classlib jars, natives, includes, etc
for java5. And would use the "version definition" metadata to choose
the fileset, and then use process() to xform the code to the target sources.
2) For development : IDE plugin where
a) I can tell plugin that my project def/configuration is whatever,
it using metadata in a file, only consider the code that is
defined (or not excluded - whatever is easier)
b) I can tell plugin to look at X, know that I want
"java5" and in situ, it shows me what Z would look like.
I edit this, but I'm really editing X. And I want to
be sure, I push a button or have a split screen that
shows me what X really looks like.
I can edit in X, or in "virtual Z", or both at the same
time, as I'm just really editing X.
I think it's key that our solution can work in an IDE, not just on a
command line.
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Mikhail Fursov wrote:
> On 11/1/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>> For comfortable IDE development, one could imagine that the IDE editor
>> can reduce to "one-line visible" comments (or better, specially
>> formatted ones) so that it gives you the impression that you are really
>> wearing target-specific spectacles. [I know Eclipse allows for such
>> things already].
>>...
> Etienne,
> What is 'comfortable IDE development' if you can't modify the Y? Am I
> missing something here?
Maybe my text was wrongly formatted... You looked at the wrong example.
Comfortable development happens only using "development targets". E.g.
1- process(X, devtarget) -> Z
2- edit Z in IDE using comfortable development, where you see a single
commented line for every hidden stream code chunk, keeping you aware
that other streams have related code there [you click on the "+" in
Eclipse if you want to see the complete chunk]. Of course, you
should never delete a chunk without consulting other stream
developers first.
So: edit Z -> Z'
3- revert(Z') -> X' this works, as long as devtarget is a stream code
preserving target (a "development target").
4- svn ci of X' :-)
You wouldn't want to do the same with a "release target". A release
target is a target where you want to completely remove other streams
source code from the processor output. This is so your typical J2SE
"end developer" that looks at Harmony's source.zip code wouldn't have
lots of non J2SE commented code in his face. [This was a concern
expressed earlier in this thread].
The output of process(X, releasetarget) should NOT be used for
development, not within, nor outside an IDE. It's only role is to give
end developers "clean" source code to look at. :-)
Etienne
PS: Note how "revert()" does not expect a target argument. So, to
switch from j2me to j2xx development, one must: process(revert(Z),j2xx)
[where Z == process(X,j2me)]. This simplifies quite a few things...
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Mikhail Fursov <mi...@gmail.com>.
On 11/1/06, Etienne Gagnon <eg...@sablevm.org> wrote:
>
> For comfortable IDE development, one could imagine that the IDE editor
> can reduce to "one-line visible" comments (or better, specially
> formatted ones) so that it gives you the impression that you are really
> wearing target-specific spectacles. [I know Eclipse allows for such
> things already].
>
> To release code, one would apply:
>
> process(X, release-target) => Y
>
> Now, it is important to understand that Y, in this case, is NOT suitable
> for doing any modification as
>
> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
> it won't crash.)
>
>
Etienne,
What is 'comfortable IDE development' if you can't modify the Y? Am I
missing something here?
--
Mikhail Fursov
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Cool. Any Eclipse Plugin gurus here that could create a <whatever it's
called in eclipse speak... perspective?> that has two edit panes, one
with the code w the preproc directives in it, and one that is the result
of the preproc given some defines?
geir
Nathan Beyer wrote:
> I've read a few articles about J2ME development using Antenna, which
> has a Java preprocessor task for Ant [1]. The Mobility Pack for
> NetBeans includes a Java preprocessor, which it claims is
> similar/based on Antenna [2].
>
> I still feel a little dirty about the thought of preprocessing Java
> source, but I guess it's done.
>
> -Nathan
>
> [1] http://antenna.sourceforge.net/#preprocess
> [2] http://www.netbeans.org/kb/50/quickstart-mobility.html
>
> On 11/1/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>
>>
>> Etienne Gagnon wrote:
>> > Geir Magnusson Jr. wrote:
>> >> There's caching too, I think. LogCache4J
>> >>
>> >> What I meant was that it didn't seem like we came to a conclusion
>> on it
>> >> - that if we had a general pre-processing solution, we could use that
>> >> too for logging, rather than have two.
>> >>
>> >> The actual use-cases will help figure this out.
>> >
>> > Here two typical some use cases, and some proposed solutions:
>> >
>> > Problem
>> > -------
>> > logging, and other situations where you really don't want to put the
>> > "additional" source code in the main source files
>> >
>> > Solution
>> > --------
>> > use aspects (Plug: you might want to give a look at the optimizing abc
>> > compiler)
>> >
>> >
>> > Problem
>> > -------
>> > supporting a different API specifications/versions, such as j2me and
>> > j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>> >
>> >
>> > Solution
>> > --------
>> >
>>
>> Right - these were the main hypotheticals that started this whole thread
>> off - I was thinking more about a definite example, like a given class C
>> has a different implementation of method M in Java5 than in Java6.
>>
>> Even if we walk through this with a clear example simple manual process,
>> it will be enlightening.
>>
>> > This is a trickier problem. We can divide the problem at two main
>> levels:
>> >
>> > 1- file/directory level
>> > 2- source-code level
>> >
>> > At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
>> > might include additional files (relative to the main version), and
>> might
>> > not include some files of the main version. In other words, j2me might
>> > not contain database APIs.
>>
>> Yep
>>
>> >
>> > Managing file inclusion/exclusion can be done in various ways.
>> >
>> > a) ant based: use distinct ant (sub-)files for each version.
>> >
>> > The problem is that IDEs (e.g. Eclipse) will most likely show some
>> > of the excluded files in its class/files browser. This wouldn't be
>> > very elegant. It also implies "always" compiling through ant files.
>> >
>> > Of course, one could develop Eclipse-specific configuration files to
>> > mimic the inclusion/exclusion of ant files, but then, this opens the
>> > door for discrepancies between ant vs eclipse inclusion/exclusion
>> > lists. I don't like it.
>> >
>>
>> Me neither.
>>
>> > b) custom-tool based: the custom processing tool I proposed could also
>> > parse inclusion/exclusion lists, and use these lists to move files
>> > around, in addition to processing the content of unmoved files.
>>
>> I'm not sure anything needs to be moved.
>>
>> >
>> > For example, if class X of the main version is not part of j2me,
>> > "process(j2me)" would move this file to a subdirectory ".streams/".
>> >
>> > If a class Y is not in the "main" version (the one used for "svn
>> > ci"), it resides in subdirectory ".streams" in the trunk.
>> > "process(j2me)" moves this file into the normal directory.
>> >
>> > As for IDEs, now you can configure them to automatically exclude
>> > ".stream/" directories.
>>
>> This would get messy. I'd rather just have a plug-in that reads a
>> description file for version X and just doesn't show me what isn't part
>> of version X.
>>
>> >
>> > Inclusion/exclusion could be managed in two ways:
>> > 1- the processing tool could look for inclusion/exclusion list
>> files,
>> > such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>> > main.exclusion, etc."
>> >
>> > This would lead to the best performance (for process(X)), yet it
>> > does require much manual update of inclusion/exclusion lists,
>> with
>> > all the discrepancies that result from human mistakes while
>> > updating these files.
>> >
>> > 2- (my preferred way), directives, at the beginning of the source
>> > code would indicate whether a file is included in a version or
>> > not. Depending on target, the file would be moved to the
>> > appropriate directory (normal, ".streams").
>>
>> Either one - the nice thing about #1 over #2 is that you can actually
>> look at it to get a summary. But you can generate the same info w/ a
>> tree walk, I suppose.
>>
>> >
>> >
>> > Of course, there's also the problem of non-source files, i.e.
>> > resources. IMO, resources could be managed using specific
>> > directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
>> > directory with symbolic links in the specific directories.
>> >
>>
>> This is getting messy.
>>
>> >
>> > As for source-level management, you would use my proposed processing
>> > tool to view the source files with the right spectacles [as Tim said so
>> > elegantly:-)]. For "development targets", it is important that:
>> >
>> > revert(process(X, target)) => X
>> >
>> > By "development target" I mean a target that is meant for Harmony
>> > developers in prevision of reverting "modified code" to a format
>> > suitable for "svn ci" (i.e. revert to "main" target).
>> >
>> > For comfortable IDE development, one could imagine that the IDE editor
>> > can reduce to "one-line visible" comments (or better, specially
>> > formatted ones) so that it gives you the impression that you are really
>> > wearing target-specific spectacles. [I know Eclipse allows for such
>> > things already].
>> >
>> > To release code, one would apply:
>> >
>> > process(X, release-target) => Y
>> >
>> > Now, it is important to understand that Y, in this case, is NOT
>> suitable
>> > for doing any modification as
>> >
>> > revert(Y) => Kaboom! (The tool will simply report that it can't do
>> it;
>> > it won't crash.)
>> >
>> > Yet, I think that it would be important that the processing tool leaves
>> > "markers" in Y, so that we could also have a tool to help finding the
>> > canonical source line of a reported bug:
>> >
>> > revertLine(Y, L') => L (where L' is a line reported by end-developer,
>> > and L the related line in "svn").
>> >
>> > Markers would be short single lines comments, so the end-developer
>> > annoyance would be kept minimal.
>> >
>> >
>> > What do you think?
>>
>> This is what I thought we were talking about all along - basically
>> starting w/ the full source, and pre-process to the "canonical" source
>> for the target version.
>>
>> However, I don't understand why I can't go backwards, modulo some manual
>> merging if needed.
>>
>> For example, if I have X and release-version, I should be able to take a
>> Y' and resolve back to X'. There are problematic cases. For example,
>> in psuedo code and psuedo directives :
>>
>> X :
>>
>> public void woogie() {
>>
>> firstLineJava;
>>
>> #ifdef(Java 6)
>> middleLineJava6;
>> #endif
>>
>> lastLineJava;
>> }
>>
>> then
>>
>> process(X, Java5) outputs
>>
>>
>> Y:
>>
>> public void woogie() {
>>
>> firstLineJava;
>>
>> lastLineJava;
>> }
>>
>>
>> Now, if I make a Y'
>>
>> public void woogie() {
>> newLineJava5;
>>
>> firstLineJava;
>>
>> lastLineJava;
>> }
>>
>> I should be able to
>>
>> revert(X, java5, Y') -> X' such that
>>
>> X' :
>>
>> public void woogie() {
>>
>> #ifdef(java5)
>> newLineJava5;
>> #endif
>>
>> firstLineJava;
>>
>> #ifdef(Java 6)
>> middleLineJava6;
>> #endif
>>
>> lastLineJava;
>> }
>>
>> No problem. (Maybe a flag that tells revert that the change is for all
>> versions....)
>>
>> However if
>>
>> Y' :
>>
>> public void woogie() {
>>
>> firstLineJava;
>>
>> newLineJava5;
>>
>> lastLineJava;
>> }
>>
>> then I'm not sure if X' is
>>
>> public void woogie() {
>>
>> firstLineJava;
>>
>> #ifdef(java5)
>> newLineJava5;
>> #endif
>>
>> #ifdef(Java 6)
>> middleLineJava6;
>> #endif
>>
>> lastLineJava;
>> }
>>
>> or
>>
>> public void woogie() {
>>
>> firstLineJava;
>>
>> #ifdef(Java 6)
>> middleLineJava6;
>> #endif
>>
>> #ifdef(java5)
>> newLineJava5;
>> #endif
>>
>> lastLineJava;
>> }
>>
>> In this case it may not matter, but either way, simply doing a
>> merge-failure style >>>>>> marker should make it clear to the developer
>> that some assistance is needed.
>>
>> I also realize that this is a toy simplistic case :)
>>
>> >
>> >
>> > I am really offering to develop this custom tool. Just help me
>> identify
>> > various Harmony needs I might have missed!
>>
>> Cool! Go for it!
>>
>> >
>> > Of course, this tool is not the best solution to ALL problems, yet, so
>> > far, I think that it seems to best address the problem of supporting
>> > various API versions.
>>
>> lets sketch out a few real examples, say between Java5 and Java6, and
>> work them through. That should give you a good idea on what the tool
>> will be required to do.
>>
>> geir
>>
>>
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Nathan Beyer <nb...@gmail.com>.
I've read a few articles about J2ME development using Antenna, which
has a Java preprocessor task for Ant [1]. The Mobility Pack for
NetBeans includes a Java preprocessor, which it claims is
similar/based on Antenna [2].
I still feel a little dirty about the thought of preprocessing Java
source, but I guess it's done.
-Nathan
[1] http://antenna.sourceforge.net/#preprocess
[2] http://www.netbeans.org/kb/50/quickstart-mobility.html
On 11/1/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
> Etienne Gagnon wrote:
> > Geir Magnusson Jr. wrote:
> >> There's caching too, I think. LogCache4J
> >>
> >> What I meant was that it didn't seem like we came to a conclusion on it
> >> - that if we had a general pre-processing solution, we could use that
> >> too for logging, rather than have two.
> >>
> >> The actual use-cases will help figure this out.
> >
> > Here two typical some use cases, and some proposed solutions:
> >
> > Problem
> > -------
> > logging, and other situations where you really don't want to put the
> > "additional" source code in the main source files
> >
> > Solution
> > --------
> > use aspects (Plug: you might want to give a look at the optimizing abc
> > compiler)
> >
> >
> > Problem
> > -------
> > supporting a different API specifications/versions, such as j2me and
> > j2se1.4, in addition to the "main" version (e.g. j2se1.5)
> >
> >
> > Solution
> > --------
> >
>
> Right - these were the main hypotheticals that started this whole thread
> off - I was thinking more about a definite example, like a given class C
> has a different implementation of method M in Java5 than in Java6.
>
> Even if we walk through this with a clear example simple manual process,
> it will be enlightening.
>
> > This is a trickier problem. We can divide the problem at two main levels:
> >
> > 1- file/directory level
> > 2- source-code level
> >
> > At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> > might include additional files (relative to the main version), and might
> > not include some files of the main version. In other words, j2me might
> > not contain database APIs.
>
> Yep
>
> >
> > Managing file inclusion/exclusion can be done in various ways.
> >
> > a) ant based: use distinct ant (sub-)files for each version.
> >
> > The problem is that IDEs (e.g. Eclipse) will most likely show some
> > of the excluded files in its class/files browser. This wouldn't be
> > very elegant. It also implies "always" compiling through ant files.
> >
> > Of course, one could develop Eclipse-specific configuration files to
> > mimic the inclusion/exclusion of ant files, but then, this opens the
> > door for discrepancies between ant vs eclipse inclusion/exclusion
> > lists. I don't like it.
> >
>
> Me neither.
>
> > b) custom-tool based: the custom processing tool I proposed could also
> > parse inclusion/exclusion lists, and use these lists to move files
> > around, in addition to processing the content of unmoved files.
>
> I'm not sure anything needs to be moved.
>
> >
> > For example, if class X of the main version is not part of j2me,
> > "process(j2me)" would move this file to a subdirectory ".streams/".
> >
> > If a class Y is not in the "main" version (the one used for "svn
> > ci"), it resides in subdirectory ".streams" in the trunk.
> > "process(j2me)" moves this file into the normal directory.
> >
> > As for IDEs, now you can configure them to automatically exclude
> > ".stream/" directories.
>
> This would get messy. I'd rather just have a plug-in that reads a
> description file for version X and just doesn't show me what isn't part
> of version X.
>
> >
> > Inclusion/exclusion could be managed in two ways:
> > 1- the processing tool could look for inclusion/exclusion list files,
> > such as "j2me.inclusion, j2me.exclusion, main.inclusion,
> > main.exclusion, etc."
> >
> > This would lead to the best performance (for process(X)), yet it
> > does require much manual update of inclusion/exclusion lists, with
> > all the discrepancies that result from human mistakes while
> > updating these files.
> >
> > 2- (my preferred way), directives, at the beginning of the source
> > code would indicate whether a file is included in a version or
> > not. Depending on target, the file would be moved to the
> > appropriate directory (normal, ".streams").
>
> Either one - the nice thing about #1 over #2 is that you can actually
> look at it to get a summary. But you can generate the same info w/ a
> tree walk, I suppose.
>
> >
> >
> > Of course, there's also the problem of non-source files, i.e.
> > resources. IMO, resources could be managed using specific
> > directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> > directory with symbolic links in the specific directories.
> >
>
> This is getting messy.
>
> >
> > As for source-level management, you would use my proposed processing
> > tool to view the source files with the right spectacles [as Tim said so
> > elegantly:-)]. For "development targets", it is important that:
> >
> > revert(process(X, target)) => X
> >
> > By "development target" I mean a target that is meant for Harmony
> > developers in prevision of reverting "modified code" to a format
> > suitable for "svn ci" (i.e. revert to "main" target).
> >
> > For comfortable IDE development, one could imagine that the IDE editor
> > can reduce to "one-line visible" comments (or better, specially
> > formatted ones) so that it gives you the impression that you are really
> > wearing target-specific spectacles. [I know Eclipse allows for such
> > things already].
> >
> > To release code, one would apply:
> >
> > process(X, release-target) => Y
> >
> > Now, it is important to understand that Y, in this case, is NOT suitable
> > for doing any modification as
> >
> > revert(Y) => Kaboom! (The tool will simply report that it can't do it;
> > it won't crash.)
> >
> > Yet, I think that it would be important that the processing tool leaves
> > "markers" in Y, so that we could also have a tool to help finding the
> > canonical source line of a reported bug:
> >
> > revertLine(Y, L') => L (where L' is a line reported by end-developer,
> > and L the related line in "svn").
> >
> > Markers would be short single lines comments, so the end-developer
> > annoyance would be kept minimal.
> >
> >
> > What do you think?
>
> This is what I thought we were talking about all along - basically
> starting w/ the full source, and pre-process to the "canonical" source
> for the target version.
>
> However, I don't understand why I can't go backwards, modulo some manual
> merging if needed.
>
> For example, if I have X and release-version, I should be able to take a
> Y' and resolve back to X'. There are problematic cases. For example,
> in psuedo code and psuedo directives :
>
> X :
>
> public void woogie() {
>
> firstLineJava;
>
> #ifdef(Java 6)
> middleLineJava6;
> #endif
>
> lastLineJava;
> }
>
> then
>
> process(X, Java5) outputs
>
>
> Y:
>
> public void woogie() {
>
> firstLineJava;
>
> lastLineJava;
> }
>
>
> Now, if I make a Y'
>
> public void woogie() {
> newLineJava5;
>
> firstLineJava;
>
> lastLineJava;
> }
>
> I should be able to
>
> revert(X, java5, Y') -> X' such that
>
> X' :
>
> public void woogie() {
>
> #ifdef(java5)
> newLineJava5;
> #endif
>
> firstLineJava;
>
> #ifdef(Java 6)
> middleLineJava6;
> #endif
>
> lastLineJava;
> }
>
> No problem. (Maybe a flag that tells revert that the change is for all
> versions....)
>
> However if
>
> Y' :
>
> public void woogie() {
>
> firstLineJava;
>
> newLineJava5;
>
> lastLineJava;
> }
>
> then I'm not sure if X' is
>
> public void woogie() {
>
> firstLineJava;
>
> #ifdef(java5)
> newLineJava5;
> #endif
>
> #ifdef(Java 6)
> middleLineJava6;
> #endif
>
> lastLineJava;
> }
>
> or
>
> public void woogie() {
>
> firstLineJava;
>
> #ifdef(Java 6)
> middleLineJava6;
> #endif
>
> #ifdef(java5)
> newLineJava5;
> #endif
>
> lastLineJava;
> }
>
> In this case it may not matter, but either way, simply doing a
> merge-failure style >>>>>> marker should make it clear to the developer
> that some assistance is needed.
>
> I also realize that this is a toy simplistic case :)
>
> >
> >
> > I am really offering to develop this custom tool. Just help me identify
> > various Harmony needs I might have missed!
>
> Cool! Go for it!
>
> >
> > Of course, this tool is not the best solution to ALL problems, yet, so
> > far, I think that it seems to best address the problem of supporting
> > various API versions.
>
> lets sketch out a few real examples, say between Java5 and Java6, and
> work them through. That should give you a good idea on what the tool
> will be required to do.
>
> geir
>
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>>> To release code, one would apply:
>>>
>>> process(X, release-target) => Y
>>>
>>> Now, it is important to understand that Y, in this case, is NOT suitable
>>> for doing any modification as
>>>
>>> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
>>> it won't crash.)
>>> ...
>> This is what I thought we were talking about all along - basically
>> starting w/ the full source, and pre-process to the "canonical" source
>> for the target version.
>>
>> However, I don't understand why I can't go backwards, modulo some manual
>> merging if needed.
>>
>> For example, if I have X and release-version, I should be able to take a
>> Y' and resolve back to X'. There are problematic cases. [...]
>
> It's the problematic cases I am trying to get entirely rid of "by
> design", by having an "exact" transformation process, in contrast with
> the usual branch/merge "inexact" process.
Yes, this is the key point for me, that the transforms are exactly
reversible so you loose nothing by switching back and forth.
> Of course, this means that development happens using "complete" source
> code (this is either the canonical form or a development target form,
> but not a release form).
Ack, and I agree with the comment made elsewhere that it is useful for
each processed form's developer to see their changes in the context of
all potential processed forms.
> For applying user-provided patches, then it can still be done the good
> old way, using these "little" patches made from release code, but
> applying them directly to the canonical or dev-target code, resolving
> conflicts manually. [The smaller the patch, the easier it is to apply
> it on files with target-specific code.]
How so? With a customized patch tool? Because the end-user's patch
will have line numbers relative to the 'releasetarget' code.
> The idea is that manual processing is never required when coding using
> canonical and dev-target modes. No inexact modifications, no difficult
> merges, etc.
Hurray.
> And, as a bonus, communication between parrallel target
> developers. I know, it's an "unusual" development processing tool, but,
> hey, why shouldn't Harmony innovate? :-)
...continue to innovate ;-)
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>>> To release code, one would apply:
>>>
>>> process(X, release-target) => Y
>>>
>>> Now, it is important to understand that Y, in this case, is NOT suitable
>>> for doing any modification as
>>>
>>> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
>>> it won't crash.)
>>> ...
>> This is what I thought we were talking about all along - basically
>> starting w/ the full source, and pre-process to the "canonical" source
>> for the target version.
>>
>> However, I don't understand why I can't go backwards, modulo some manual
>> merging if needed.
>>
>> For example, if I have X and release-version, I should be able to take a
>> Y' and resolve back to X'. There are problematic cases. [...]
>
>
> It's the problematic cases I am trying to get entirely rid of "by
> design", by having an "exact" transformation process, in contrast with
> the usual branch/merge "inexact" process.
At what cost though? We are used to the inexact process now...
>
> Of course, this means that development happens using "complete" source
> code (this is either the canonical form or a development target form,
> but not a release form).
What's the diff?
>
> For applying user-provided patches, then it can still be done the good
> old way, using these "little" patches made from release code, but
> applying them directly to the canonical or dev-target code, resolving
> conflicts manually. [The smaller the patch, the easier it is to apply
> it on files with target-specific code.]
In which case we have the inexact process anyway.
>
> The idea is that manual processing is never required when coding using
> canonical and dev-target modes. No inexact modifications, no difficult
> merges, etc. And, as a bonus, communication between parrallel target
> developers. I know, it's an "unusual" development processing tool, but,
> hey, why shouldn't Harmony innovate? :-)
I'm missing something then. You just said you can't go backwards.
There is only one file for class C in SVN. How do I work on anything
but that copy if there is no going backwards?
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
>> To release code, one would apply:
>>
>> process(X, release-target) => Y
>>
>> Now, it is important to understand that Y, in this case, is NOT suitable
>> for doing any modification as
>>
>> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
>> it won't crash.)
>>...
>
> This is what I thought we were talking about all along - basically
> starting w/ the full source, and pre-process to the "canonical" source
> for the target version.
>
> However, I don't understand why I can't go backwards, modulo some manual
> merging if needed.
>
> For example, if I have X and release-version, I should be able to take a
> Y' and resolve back to X'. There are problematic cases. [...]
It's the problematic cases I am trying to get entirely rid of "by
design", by having an "exact" transformation process, in contrast with
the usual branch/merge "inexact" process.
Of course, this means that development happens using "complete" source
code (this is either the canonical form or a development target form,
but not a release form).
For applying user-provided patches, then it can still be done the good
old way, using these "little" patches made from release code, but
applying them directly to the canonical or dev-target code, resolving
conflicts manually. [The smaller the patch, the easier it is to apply
it on files with target-specific code.]
The idea is that manual processing is never required when coding using
canonical and dev-target modes. No inexact modifications, no difficult
merges, etc. And, as a bonus, communication between parrallel target
developers. I know, it's an "unusual" development processing tool, but,
hey, why shouldn't Harmony innovate? :-)
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> There's caching too, I think. LogCache4J
>>
>> What I meant was that it didn't seem like we came to a conclusion on it
>> - that if we had a general pre-processing solution, we could use that
>> too for logging, rather than have two.
>>
>> The actual use-cases will help figure this out.
>
> Here two typical some use cases, and some proposed solutions:
>
> Problem
> -------
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
>
> Solution
> --------
> use aspects (Plug: you might want to give a look at the optimizing abc
> compiler)
>
>
> Problem
> -------
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
> Solution
> --------
>
Right - these were the main hypotheticals that started this whole thread
off - I was thinking more about a definite example, like a given class C
has a different implementation of method M in Java5 than in Java6.
Even if we walk through this with a clear example simple manual process,
it will be enlightening.
> This is a trickier problem. We can divide the problem at two main levels:
>
> 1- file/directory level
> 2- source-code level
>
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version. In other words, j2me might
> not contain database APIs.
Yep
>
> Managing file inclusion/exclusion can be done in various ways.
>
> a) ant based: use distinct ant (sub-)files for each version.
>
> The problem is that IDEs (e.g. Eclipse) will most likely show some
> of the excluded files in its class/files browser. This wouldn't be
> very elegant. It also implies "always" compiling through ant files.
>
> Of course, one could develop Eclipse-specific configuration files to
> mimic the inclusion/exclusion of ant files, but then, this opens the
> door for discrepancies between ant vs eclipse inclusion/exclusion
> lists. I don't like it.
>
Me neither.
> b) custom-tool based: the custom processing tool I proposed could also
> parse inclusion/exclusion lists, and use these lists to move files
> around, in addition to processing the content of unmoved files.
I'm not sure anything needs to be moved.
>
> For example, if class X of the main version is not part of j2me,
> "process(j2me)" would move this file to a subdirectory ".streams/".
>
> If a class Y is not in the "main" version (the one used for "svn
> ci"), it resides in subdirectory ".streams" in the trunk.
> "process(j2me)" moves this file into the normal directory.
>
> As for IDEs, now you can configure them to automatically exclude
> ".stream/" directories.
This would get messy. I'd rather just have a plug-in that reads a
description file for version X and just doesn't show me what isn't part
of version X.
>
> Inclusion/exclusion could be managed in two ways:
> 1- the processing tool could look for inclusion/exclusion list files,
> such as "j2me.inclusion, j2me.exclusion, main.inclusion,
> main.exclusion, etc."
>
> This would lead to the best performance (for process(X)), yet it
> does require much manual update of inclusion/exclusion lists, with
> all the discrepancies that result from human mistakes while
> updating these files.
>
> 2- (my preferred way), directives, at the beginning of the source
> code would indicate whether a file is included in a version or
> not. Depending on target, the file would be moved to the
> appropriate directory (normal, ".streams").
Either one - the nice thing about #1 over #2 is that you can actually
look at it to get a summary. But you can generate the same info w/ a
tree walk, I suppose.
>
>
> Of course, there's also the problem of non-source files, i.e.
> resources. IMO, resources could be managed using specific
> directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> directory with symbolic links in the specific directories.
>
This is getting messy.
>
> As for source-level management, you would use my proposed processing
> tool to view the source files with the right spectacles [as Tim said so
> elegantly:-)]. For "development targets", it is important that:
>
> revert(process(X, target)) => X
>
> By "development target" I mean a target that is meant for Harmony
> developers in prevision of reverting "modified code" to a format
> suitable for "svn ci" (i.e. revert to "main" target).
>
> For comfortable IDE development, one could imagine that the IDE editor
> can reduce to "one-line visible" comments (or better, specially
> formatted ones) so that it gives you the impression that you are really
> wearing target-specific spectacles. [I know Eclipse allows for such
> things already].
>
> To release code, one would apply:
>
> process(X, release-target) => Y
>
> Now, it is important to understand that Y, in this case, is NOT suitable
> for doing any modification as
>
> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
> it won't crash.)
>
> Yet, I think that it would be important that the processing tool leaves
> "markers" in Y, so that we could also have a tool to help finding the
> canonical source line of a reported bug:
>
> revertLine(Y, L') => L (where L' is a line reported by end-developer,
> and L the related line in "svn").
>
> Markers would be short single lines comments, so the end-developer
> annoyance would be kept minimal.
>
>
> What do you think?
This is what I thought we were talking about all along - basically
starting w/ the full source, and pre-process to the "canonical" source
for the target version.
However, I don't understand why I can't go backwards, modulo some manual
merging if needed.
For example, if I have X and release-version, I should be able to take a
Y' and resolve back to X'. There are problematic cases. For example,
in psuedo code and psuedo directives :
X :
public void woogie() {
firstLineJava;
#ifdef(Java 6)
middleLineJava6;
#endif
lastLineJava;
}
then
process(X, Java5) outputs
Y:
public void woogie() {
firstLineJava;
lastLineJava;
}
Now, if I make a Y'
public void woogie() {
newLineJava5;
firstLineJava;
lastLineJava;
}
I should be able to
revert(X, java5, Y') -> X' such that
X' :
public void woogie() {
#ifdef(java5)
newLineJava5;
#endif
firstLineJava;
#ifdef(Java 6)
middleLineJava6;
#endif
lastLineJava;
}
No problem. (Maybe a flag that tells revert that the change is for all
versions....)
However if
Y' :
public void woogie() {
firstLineJava;
newLineJava5;
lastLineJava;
}
then I'm not sure if X' is
public void woogie() {
firstLineJava;
#ifdef(java5)
newLineJava5;
#endif
#ifdef(Java 6)
middleLineJava6;
#endif
lastLineJava;
}
or
public void woogie() {
firstLineJava;
#ifdef(Java 6)
middleLineJava6;
#endif
#ifdef(java5)
newLineJava5;
#endif
lastLineJava;
}
In this case it may not matter, but either way, simply doing a
merge-failure style >>>>>> marker should make it clear to the developer
that some assistance is needed.
I also realize that this is a toy simplistic case :)
>
>
> I am really offering to develop this custom tool. Just help me identify
> various Harmony needs I might have missed!
Cool! Go for it!
>
> Of course, this tool is not the best solution to ALL problems, yet, so
> far, I think that it seems to best address the problem of supporting
> various API versions.
lets sketch out a few real examples, say between Java5 and Java6, and
work them through. That should give you a good idea on what the tool
will be required to do.
geir
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Etienne Gagnon wrote:
>> I'm proposing to simply use "precise svn url/version identification" as
>> markup:
> <snip>
>> As long as we are *sure* that the releasetarget code was built from
>> precisely that url/version (without any "local workspace" modification)
>> then it should be as precise as having markups everywhere in the source
>> code. :-)
>
> Ah! penny has dropped! -- cool, yes. The immutability of the SVN
> revision means that you don't need to reverse the releasetarget, you can
> go back directly to the [canonical|master|stored] form and see exactly
> where the releasetarget lines came from. I like it.
>
>> Of course, this adds a (small?) dependence on Subversion, but the gains
>> in markup noise reduction seems to be worth the trouble.
>
> Yep -- it keeps the release code squeaky clean, and it's not like we
> don't depend on Subversion being there already for fixing bugs reported
> in released code<g>.
Right - we already have similar information in patches, although we
don't rely on it.
Index: index.xml
===================================================================
--- index.xml (revision 470768)
+++ index.xml (working copy)
>
> Regards,
> Tim
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> I'm proposing to simply use "precise svn url/version identification" as
> markup:
<snip>
> As long as we are *sure* that the releasetarget code was built from
> precisely that url/version (without any "local workspace" modification)
> then it should be as precise as having markups everywhere in the source
> code. :-)
Ah! penny has dropped! -- cool, yes. The immutability of the SVN
revision means that you don't need to reverse the releasetarget, you can
go back directly to the [canonical|master|stored] form and see exactly
where the releasetarget lines came from. I like it.
> Of course, this adds a (small?) dependence on Subversion, but the gains
> in markup noise reduction seems to be worth the trouble.
Yep -- it keeps the release code squeaky clean, and it's not like we
don't depend on Subversion being there already for fixing bugs reported
in released code<g>.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Tim Ellison wrote:
>>>Agreed. It would be interesting to determine the most effective
>>>location for those markers (measured by reverse mapping accuracy vs.
>>>number of markers).
>>
>>I am a fan of accuracy... Yet, maybe it would be simple enough if a
>>release was always based on a specific svn URL, then the mark could be
>>totally exact using svn keywords... :-)
>
>
> You lost me here. I'm trying to define tie-points between the
> 'releasetarget' source and the canonical form. AIUI this will require
> structured comments in the 'releasetarget', right?
Heh! I'm proposing another "unusual" markup that would be least intrusive.
I'm proposing to simply use "precise svn url/version identification" as
markup:
file:XXX.java
-------------
/*
* @ProcessIdentification(j2se,release)
* @ProcessURL(https://svn.apache.org/repos/harmony/somepath/XXX.java)
* @ProcessRevision(144)
*
* This tells us that this file is the result of:
* process(svn cat -r revision url, j2serelease)
*/
/* releasetarget code, WITHOUT any additional markup:-) */
...
...
As long as we are *sure* that the releasetarget code was built from
precisely that url/version (without any "local workspace" modification)
then it should be as precise as having markups everywhere in the source
code. :-)
Of course, this adds a (small?) dependence on Subversion, but the gains
in markup noise reduction seems to be worth the trouble.
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Tim Ellison wrote:
>>> For example, if class X of the main version is not part of j2me,
>>> "process(j2me)" would move this file to a subdirectory ".streams/".
>> Why would you move the files rather than exclude them?
>>
>> I was assuming that the processor would generate a whole new source tree
>> for each process() target, so that you could work on the original
>> checked-out file in it's 'canonicalized form' for Big Java work, or
>> process("jme") into a new location and work in the non-canonical form
>> your Little Java spectacles on. [...]
>
> Neat idea!
>
> I would put the following restriction, though: one should NOT modify
> more than one target at a time.
Fair enough -- at least for now.
> So, you would probably need some way to prevent parallel modifications
> in "spectacle views".
Perhaps we should agree to use your 'devtarget' term for this :-)
> One way to achieve this:
>
> process(X,target, destination) => Xtarget in a distinct location
> and X files are changed to read-only and some tracking file F tells us
> about Xtarget's location.
Sure, should be simple enough if there is a well-known location for this
lock file. As you say, then developers would work on a single devtarget
at a time in their local workspace, but potentially go back to the
canonical form and switch to another devtarget to check that before
checking in the canonical form.
> So, you would also need:
>
> release(F) => asks all kind of questions (want to lose changes? delete
> files? etc.)
>
> Of course:
>
> revert(F?|destination?) => makes X read-write.
>
>
> [Hoping this was clear enough... It's not a very good explanation...]
Yep, I got it; and it seems quite simple with these restrictions.
>> Agreed. It would be interesting to determine the most effective
>> location for those markers (measured by reverse mapping accuracy vs.
>> number of markers).
>
> I am a fan of accuracy... Yet, maybe it would be simple enough if a
> release was always based on a specific svn URL, then the mark could be
> totally exact using svn keywords... :-)
You lost me here. I'm trying to define tie-points between the
'releasetarget' source and the canonical form. AIUI this will require
structured comments in the 'releasetarget', right?
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Tim Ellison wrote:
>> For example, if class X of the main version is not part of j2me,
>> "process(j2me)" would move this file to a subdirectory ".streams/".
>
> Why would you move the files rather than exclude them?
>
> I was assuming that the processor would generate a whole new source tree
> for each process() target, so that you could work on the original
> checked-out file in it's 'canonicalized form' for Big Java work, or
> process("jme") into a new location and work in the non-canonical form
> your Little Java spectacles on. [...]
Neat idea!
I would put the following restriction, though: one should NOT modify
more than one target at a time.
So, you would probably need some way to prevent parallel modifications
in "spectacle views". One way to achieve this:
process(X,target, destination) => Xtarget in a distinct location
and X files are changed to read-only and some tracking file F tells us
about Xtarget's location.
So, you would also need:
release(F) => asks all kind of questions (want to lose changes? delete
files? etc.)
Of course:
revert(F?|destination?) => makes X read-write.
[Hoping this was clear enough... It's not a very good explanation...]
> Agreed. It would be interesting to determine the most effective
> location for those markers (measured by reverse mapping accuracy vs.
> number of markers).
I am a fan of accuracy... Yet, maybe it would be simple enough if a
release was always based on a specific svn URL, then the mark could be
totally exact using svn keywords... :-)
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Here two typical some use cases, and some proposed solutions:
>
> Problem
> -------
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
>
> Solution
> --------
> use aspects (Plug: you might want to give a look at the optimizing abc
> compiler)
+1 and/or 'intrinsic' VM-level tracing of the Java code.
> Problem
> -------
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
>
>
> Solution
> --------
>
> This is a trickier problem. We can divide the problem at two main levels:
>
> 1- file/directory level
> 2- source-code level
>
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version. In other words, j2me might
> not contain database APIs.
>
> Managing file inclusion/exclusion can be done in various ways.
>
> a) ant based: use distinct ant (sub-)files for each version.
>
> The problem is that IDEs (e.g. Eclipse) will most likely show some
> of the excluded files in its class/files browser. This wouldn't be
> very elegant. It also implies "always" compiling through ant files.
>
> Of course, one could develop Eclipse-specific configuration files to
> mimic the inclusion/exclusion of ant files, but then, this opens the
> door for discrepancies between ant vs eclipse inclusion/exclusion
> lists. I don't like it.
>
> b) custom-tool based: the custom processing tool I proposed could also
> parse inclusion/exclusion lists, and use these lists to move files
> around, in addition to processing the content of unmoved files.
>
> For example, if class X of the main version is not part of j2me,
> "process(j2me)" would move this file to a subdirectory ".streams/".
Why would you move the files rather than exclude them?
I was assuming that the processor would generate a whole new source tree
for each process() target, so that you could work on the original
checked-out file in it's 'canonicalized form' for Big Java work, or
process("jme") into a new location and work in the non-canonical form
your Little Java spectacles on. That would prevent you accidentally
checking in the Little Java code, since you would need to
reverse-process the code back to replace the checked-out file (with a
test to ensure you are not overwriting changes made to Big Java, or for
extra credit, merging changes in the working copies). The svn check-in
would then always be in the Big Java canonicalized form.
This idea would loose the ability to check-in directly from the IDE
though for people working with the non-canonicalized source form.
> If a class Y is not in the "main" version (the one used for "svn
> ci"), it resides in subdirectory ".streams" in the trunk.
> "process(j2me)" moves this file into the normal directory.
>
> As for IDEs, now you can configure them to automatically exclude
> ".stream/" directories.
>
> Inclusion/exclusion could be managed in two ways:
> 1- the processing tool could look for inclusion/exclusion list files,
> such as "j2me.inclusion, j2me.exclusion, main.inclusion,
> main.exclusion, etc."
>
> This would lead to the best performance (for process(X)), yet it
> does require much manual update of inclusion/exclusion lists, with
> all the discrepancies that result from human mistakes while
> updating these files.
>
> 2- (my preferred way), directives, at the beginning of the source
> code would indicate whether a file is included in a version or
> not. Depending on target, the file would be moved to the
> appropriate directory (normal, ".streams").
Agreed. Since we are requiring the source to be processed for any
deliverable target, we might as well keep the mark-up in-line.
> Of course, there's also the problem of non-source files, i.e.
> resources. IMO, resources could be managed using specific
> directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
> directory with symbolic links in the specific directories.
>
>
> As for source-level management, you would use my proposed processing
> tool to view the source files with the right spectacles [as Tim said so
> elegantly:-)]. For "development targets", it is important that:
>
> revert(process(X, target)) => X
>
> By "development target" I mean a target that is meant for Harmony
> developers in prevision of reverting "modified code" to a format
> suitable for "svn ci" (i.e. revert to "main" target).
>
> For comfortable IDE development, one could imagine that the IDE editor
> can reduce to "one-line visible" comments (or better, specially
> formatted ones) so that it gives you the impression that you are really
> wearing target-specific spectacles. [I know Eclipse allows for such
> things already].
>
> To release code, one would apply:
>
> process(X, release-target) => Y
>
> Now, it is important to understand that Y, in this case, is NOT suitable
> for doing any modification as
>
> revert(Y) => Kaboom! (The tool will simply report that it can't do it;
> it won't crash.)
>
> Yet, I think that it would be important that the processing tool leaves
> "markers" in Y, so that we could also have a tool to help finding the
> canonical source line of a reported bug:
>
> revertLine(Y, L') => L (where L' is a line reported by end-developer,
> and L the related line in "svn").
>
> Markers would be short single lines comments, so the end-developer
> annoyance would be kept minimal.
Agreed. It would be interesting to determine the most effective
location for those markers (measured by reverse mapping accuracy vs.
number of markers).
I can imagine a simple scheme by choosing the start of major Java
elements (class/method definitions), or a fancier scheme based on the
'distance' that lines in the two sources get out of sync'; e.g. if the
lines in the processed source are all within +/-5 lines of the original
then no markers are required, but if the processed source is > 5 lines
out then lay down a re-sync' point that ties them back together (e.g.
processed output line 30 is "/* Canonical Line 42 */") and watch for
further drifting from there.
> What do you think?
>
>
> I am really offering to develop this custom tool. Just help me identify
> various Harmony needs I might have missed!
Wonderful! Send in a basic tool and I'd be delighted to help shape it.
> Of course, this tool is not the best solution to ALL problems, yet, so
> far, I think that it seems to best address the problem of supporting
> various API versions.
Agreed. I'm interested to hear other people's ideas / requirements too.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Geir Magnusson Jr. wrote:
> There's caching too, I think. LogCache4J
>
> What I meant was that it didn't seem like we came to a conclusion on it
> - that if we had a general pre-processing solution, we could use that
> too for logging, rather than have two.
>
> The actual use-cases will help figure this out.
Here two typical some use cases, and some proposed solutions:
Problem
-------
logging, and other situations where you really don't want to put the
"additional" source code in the main source files
Solution
--------
use aspects (Plug: you might want to give a look at the optimizing abc
compiler)
Problem
-------
supporting a different API specifications/versions, such as j2me and
j2se1.4, in addition to the "main" version (e.g. j2se1.5)
Solution
--------
This is a trickier problem. We can divide the problem at two main levels:
1- file/directory level
2- source-code level
At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
might include additional files (relative to the main version), and might
not include some files of the main version. In other words, j2me might
not contain database APIs.
Managing file inclusion/exclusion can be done in various ways.
a) ant based: use distinct ant (sub-)files for each version.
The problem is that IDEs (e.g. Eclipse) will most likely show some
of the excluded files in its class/files browser. This wouldn't be
very elegant. It also implies "always" compiling through ant files.
Of course, one could develop Eclipse-specific configuration files to
mimic the inclusion/exclusion of ant files, but then, this opens the
door for discrepancies between ant vs eclipse inclusion/exclusion
lists. I don't like it.
b) custom-tool based: the custom processing tool I proposed could also
parse inclusion/exclusion lists, and use these lists to move files
around, in addition to processing the content of unmoved files.
For example, if class X of the main version is not part of j2me,
"process(j2me)" would move this file to a subdirectory ".streams/".
If a class Y is not in the "main" version (the one used for "svn
ci"), it resides in subdirectory ".streams" in the trunk.
"process(j2me)" moves this file into the normal directory.
As for IDEs, now you can configure them to automatically exclude
".stream/" directories.
Inclusion/exclusion could be managed in two ways:
1- the processing tool could look for inclusion/exclusion list files,
such as "j2me.inclusion, j2me.exclusion, main.inclusion,
main.exclusion, etc."
This would lead to the best performance (for process(X)), yet it
does require much manual update of inclusion/exclusion lists, with
all the discrepancies that result from human mistakes while
updating these files.
2- (my preferred way), directives, at the beginning of the source
code would indicate whether a file is included in a version or
not. Depending on target, the file would be moved to the
appropriate directory (normal, ".streams").
Of course, there's also the problem of non-source files, i.e.
resources. IMO, resources could be managed using specific
directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
directory with symbolic links in the specific directories.
As for source-level management, you would use my proposed processing
tool to view the source files with the right spectacles [as Tim said so
elegantly:-)]. For "development targets", it is important that:
revert(process(X, target)) => X
By "development target" I mean a target that is meant for Harmony
developers in prevision of reverting "modified code" to a format
suitable for "svn ci" (i.e. revert to "main" target).
For comfortable IDE development, one could imagine that the IDE editor
can reduce to "one-line visible" comments (or better, specially
formatted ones) so that it gives you the impression that you are really
wearing target-specific spectacles. [I know Eclipse allows for such
things already].
To release code, one would apply:
process(X, release-target) => Y
Now, it is important to understand that Y, in this case, is NOT suitable
for doing any modification as
revert(Y) => Kaboom! (The tool will simply report that it can't do it;
it won't crash.)
Yet, I think that it would be important that the processing tool leaves
"markers" in Y, so that we could also have a tool to help finding the
canonical source line of a reported bug:
revertLine(Y, L') => L (where L' is a line reported by end-developer,
and L the related line in "svn").
Markers would be short single lines comments, so the end-developer
annoyance would be kept minimal.
What do you think?
I am really offering to develop this custom tool. Just help me identify
various Harmony needs I might have missed!
Of course, this tool is not the best solution to ALL problems, yet, so
far, I think that it seems to best address the problem of supporting
various API versions.
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Nathan Beyer wrote:
>>> What's the concern about just using the prescribed branching pattern
>>> for SVN? There are some other nice tricks like "externals" for pulling
>>> in common files into the working copies of other branches (ala the
>>> 'concurrent' code in 'standard' that's pulled into 'enhanced' on
>>> checkout).
>> Even the authors of SVN warn people away from using externals.
>
> Yeah, and a nightmare when trying to 'tag' code -- copying the link to
> HEAD is no help.
>
>>> I would propose we at least attempt to go down a path of
>>> investigating a branching.
>> We should consider everything, but I'd personally rather keep as few
>> codelines as possible.
>
> Agreed.
>
>>> Regardless, I think we need to settle on our exact requirement first,
>>> before spending too much time on looking for a solution. For example,
>>> if logging is a real requirement, but everyone agrees it can be done
>>> via instrumentation (AspectJ, java.lang.instrument, etc), then are
>>> there any other requirements that affect the actual source files
>>> internally? If not, then could all of the other requirements be
>>> fulfilled by judicious SCM use?
>>>
>>> So, I would suggest we back up a little and just layout all of the
>>> requirements first, so we can make sure everyone's in agreement about
>>> the needs.
>
> Nathan is right -- this is hypothetical now, unless (for example) we
> start on Java 6 development now.
>
>> Exactly - we need use cases (and it's not clear that the logging
>> problems have been resolved w/ aspects yet...)
>
> You're joking, right? I tease the aspect people that logging is the
> only problem that has been solved(*) <g>. There are lots of references
> on how to do that, eg:
> http://www.developer.com/java/other/article.php/3109831
There's caching too, I think. LogCache4J
What I meant was that it didn't seem like we came to a conclusion on it
- that if we had a general pre-processing solution, we could use that
too for logging, rather than have two.
The actual use-cases will help figure this out.
geir
>
>
> (*) it's not true though, there are a number of tasks that are
> well-suited to using aspects. However, I would use them judiciously.
Like caching :) (And get your local psychic retainer to tell you what
the code is doing... ;)
geir
>
> Regards,
> Tim
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr. wrote:
> Nathan Beyer wrote:
>> What's the concern about just using the prescribed branching pattern
>> for SVN? There are some other nice tricks like "externals" for pulling
>> in common files into the working copies of other branches (ala the
>> 'concurrent' code in 'standard' that's pulled into 'enhanced' on
>> checkout).
>
> Even the authors of SVN warn people away from using externals.
Yeah, and a nightmare when trying to 'tag' code -- copying the link to
HEAD is no help.
>> I would propose we at least attempt to go down a path of
>> investigating a branching.
>
> We should consider everything, but I'd personally rather keep as few
> codelines as possible.
Agreed.
>> Regardless, I think we need to settle on our exact requirement first,
>> before spending too much time on looking for a solution. For example,
>> if logging is a real requirement, but everyone agrees it can be done
>> via instrumentation (AspectJ, java.lang.instrument, etc), then are
>> there any other requirements that affect the actual source files
>> internally? If not, then could all of the other requirements be
>> fulfilled by judicious SCM use?
>>
>> So, I would suggest we back up a little and just layout all of the
>> requirements first, so we can make sure everyone's in agreement about
>> the needs.
Nathan is right -- this is hypothetical now, unless (for example) we
start on Java 6 development now.
> Exactly - we need use cases (and it's not clear that the logging
> problems have been resolved w/ aspects yet...)
You're joking, right? I tease the aspect people that logging is the
only problem that has been solved(*) <g>. There are lots of references
on how to do that, eg:
http://www.developer.com/java/other/article.php/3109831
(*) it's not true though, there are a number of tasks that are
well-suited to using aspects. However, I would use them judiciously.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Nathan Beyer wrote:
> What's the concern about just using the prescribed branching pattern
> for SVN? There are some other nice tricks like "externals" for pulling
> in common files into the working copies of other branches (ala the
> 'concurrent' code in 'standard' that's pulled into 'enhanced' on
> checkout).
Even the authors of SVN warn people away from using externals.
> I would propose we at least attempt to go down a path of
> investigating a branching.
We should consider everything, but I'd personally rather keep as few
codelines as possible.
>
> Regardless, I think we need to settle on our exact requirement first,
> before spending too much time on looking for a solution. For example,
> if logging is a real requirement, but everyone agrees it can be done
> via instrumentation (AspectJ, java.lang.instrument, etc), then are
> there any other requirements that affect the actual source files
> internally? If not, then could all of the other requirements be
> fulfilled by judicious SCM use?
>
> So, I would suggest we back up a little and just layout all of the
> requirements first, so we can make sure everyone's in agreement about
> the needs.
Exactly - we need use cases (and it's not clear that the logging
problems have been resolved w/ aspects yet...)
geir
>
> -Nathan
>
> On 10/31/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>> So we all agree that it's not an ideal solution.
>>
>> Can anyone think of anything else? No one said this was going to be
>> easy...
>>
>> geir
>>
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Nathan Beyer <nb...@gmail.com>.
What's the concern about just using the prescribed branching pattern
for SVN? There are some other nice tricks like "externals" for pulling
in common files into the working copies of other branches (ala the
'concurrent' code in 'standard' that's pulled into 'enhanced' on
checkout). I would propose we at least attempt to go down a path of
investigating a branching.
Regardless, I think we need to settle on our exact requirement first,
before spending too much time on looking for a solution. For example,
if logging is a real requirement, but everyone agrees it can be done
via instrumentation (AspectJ, java.lang.instrument, etc), then are
there any other requirements that affect the actual source files
internally? If not, then could all of the other requirements be
fulfilled by judicious SCM use?
So, I would suggest we back up a little and just layout all of the
requirements first, so we can make sure everyone's in agreement about
the needs.
-Nathan
On 10/31/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> So we all agree that it's not an ideal solution.
>
> Can anyone think of anything else? No one said this was going to be easy...
>
> geir
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Sian January wrote:
> I may be totally off track here, but how about just having two copies
> of all the files that differ? I don't believe it would be that many,
> and it would save us from having complicated source files or having to
> use special tools or special IDE plug-ins. For me the value of having
> clearly readable source code and being able to use an IDE out of the box
> outweighs any extra effort there may be with this solution.
Because I think that still means we have separate branches, and thus the
integration problem for fixes.
geir
>
> Regards,
>
> Sian
>
>
>
> On 31/10/06, *Geir Magnusson Jr.* <geir@pobox.com
> <ma...@pobox.com>> wrote:
>
> So we all agree that it's not an ideal solution.
>
> Can anyone think of anything else? No one said this was going to be
> easy...
>
> geir
>
>
>
>
> --
> Sian January
>
> IBM Java Technology Centre, UK
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Sian January wrote:
> I may be totally off track here, but how about just having two copies of
> all
> the files that differ? I don't believe it would be that many, and it would
> save us from having complicated source files or having to use special tools
> or special IDE plug-ins. For me the value of having clearly readable
> source
> code and being able to use an IDE out of the box outweighs any extra effort
> there may be with this solution.
I suspect that "two copies" might eventually become "three, four, five"
copies... Also, having parrallel copies does not help when modifying
"shared" code in these copies. Actually, it would be even worse than
using the normal branching feature of Subversion, as you wouldn't get
neat "svn merge" help...
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Sian January <si...@googlemail.com>.
I may be totally off track here, but how about just having two copies of all
the files that differ? I don't believe it would be that many, and it would
save us from having complicated source files or having to use special tools
or special IDE plug-ins. For me the value of having clearly readable source
code and being able to use an IDE out of the box outweighs any extra effort
there may be with this solution.
Regards,
Sian
On 31/10/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
> So we all agree that it's not an ideal solution.
>
> Can anyone think of anything else? No one said this was going to be
> easy...
>
> geir
>
--
Sian January
IBM Java Technology Centre, UK
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Tim Ellison wrote:
>> IMO it's not ideal that the preprocessed source still contains all the
>> streams, albeit in comments. It wouldn't make the source very
>> 'consumable' to the Mrs. SE or ME developer.
>
> Hmmm... It's always possible to have a special output mode that puts
> empty (or advertizing, hehe) comments, instead of other stream code
> (thus, preserving line numbers).
Right, but it was the presence of the comments, not the contents of the
comments that I was objecting to. Using padding comments would be bad
too IMO.
<snipped example>
> So, J2ME & J2SE end-developers are kept happy.
As I said before, I like the idea to be able to flip the comments
between different processor targets for the benefit of Harmony
developers. So, for example, you can work with your SE spectacles on,
and I can work with my ME spectacles on, then we both commit to the
repository in a canonical form. However, for the end-user (Mrs Java
developer) the processor would strip out the irrelevant streams' code
from the canonical form to produce a clean target source code.
> As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
> in $ource code. ;-P
LOL -- hey, now you are talking ;-)
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
Re: [classlib] Preprocessor - CHECKPOINT
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Etienne Gagnon wrote:
> Tim Ellison wrote:
>> IMO it's not ideal that the preprocessed source still contains all the
>> streams, albeit in comments. It wouldn't make the source very
>> 'consumable' to the Mrs. SE or ME developer.
>
> Hmmm... It's always possible to have a special output mode that puts
> empty (or advertizing, hehe) comments, instead of other stream code
> (thus, preserving line numbers).
>
> To continue on my earlier example:
>
> Java source => "j2se end-developer"
> -----------------------------------
>
>
> ...
>
> // Download Harmony[tm] from
> //
> // http://the.nice.harmony.url/download
> //
> // :-)
>
> @Processor(Not in j2me!)
> int some_field =
> some +
> initializing()
> code;
>
> ...
>
>
> Or, more likely:
>
> Java source => "j2se end-developer"
> -----------------------------------
>
>
> ...
>
> // Please ignore this comment. It has been
> // intentionally left here to preserve line numbers
> // for bug reporting purpose.
> //
> // Please report bugs to http://bugs.of.harmony.url/...
>
> @Processor(Not in j2me!)
> int some_field =
> some +
> initializing()
> code;
>
> ...
>
>
> So, J2ME & J2SE end-developers are kept happy.
I'm still not quite getting the importance of preserving the line
numbers like that if we have some minimal tooling to let us work with it
in eclipse or IDEA invisibly.... users would report bug at line X, and
we'd either look at the transformed code that they are actually using,
and translate backwards, or have a plugin that lets us "A/B" between
transformed and original.
The key is to play with some examples, I guess.
>
> As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
> in $ource code. ;-P
It's our idea - you run with it. let us know how that works out. (I
bet if you could convince some investor that it was a web2.0 thing, you
could get it funded....)
>
> Etienne
>
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
Tim Ellison wrote:
> IMO it's not ideal that the preprocessed source still contains all the
> streams, albeit in comments. It wouldn't make the source very
> 'consumable' to the Mrs. SE or ME developer.
Hmmm... It's always possible to have a special output mode that puts
empty (or advertizing, hehe) comments, instead of other stream code
(thus, preserving line numbers).
To continue on my earlier example:
Java source => "j2se end-developer"
-----------------------------------
...
// Download Harmony[tm] from
//
// http://the.nice.harmony.url/download
//
// :-)
@Processor(Not in j2me!)
int some_field =
some +
initializing()
code;
...
Or, more likely:
Java source => "j2se end-developer"
-----------------------------------
...
// Please ignore this comment. It has been
// intentionally left here to preserve line numbers
// for bug reporting purpose.
//
// Please report bugs to http://bugs.of.harmony.url/...
@Processor(Not in j2me!)
int some_field =
some +
initializing()
code;
...
So, J2ME & J2SE end-developers are kept happy.
As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace
in $ource code. ;-P
Etienne
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Tim Ellison <t....@gmail.com>.
Etienne Gagnon wrote:
> Here's an idea.
<big snip>
IMO it's not ideal that the preprocessed source still contains all the
streams, albeit in comments. It wouldn't make the source very
'consumable' to the Mrs. SE or ME developer.
The ability for classlib developers to do the switching would be good
though. At least we could talk between ourselves in common terms.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
Re: [classlib] Preprocessor - CHECKPOINT
Posted by Etienne Gagnon <eg...@sablevm.org>.
[WARNING: Long!]
Here's an idea.
It would be quite easy (for me, at least ;-) ) to develop a
"line-preserving" AND "revertible" processing tool so that:
1- source + javac => j2se
2- source + processor(target) + javac => target, where target is a
parameter (j2me, etc.)
3- source + processor(target) => source for target, where line numbers
are identical to source
4- source for target + reverseprocessor(target) -> source :-) That's
the thing lacking in most preprocessors, as often you would like to
modify the processed source and propagate these changes back to the
initial source; this is usually a pain as it must be done manually. Not
here!
I am, usually, in favor of using branches, for development, yet, in the
current case, I think that branches have their drawbacks:
1- I have yet to see a branch/merge system that deals well with major
code reorganization in a single file.
2- There's always the infrequent, yet very annoying (if not dangerous
when it happens) semantically invalid merge result.
3- Developers of the main branch will remain mostly unaware of the
parallel development on other branches. In this case, this can be a
problem, as having j2se developers aware that they are changing lines
for which other branches maintain distinct versions ones might be a good
"communication tool".
The idea, for line preservation, would be to either use "revertible"
comments (easiest) or maybe play with special Java annotations, if they
prove sufficient for the task.
The advantage of "intelligent" Java-syntax-aware processing (instead of
dumb preprocessing) is that the processor could work intelligently.
Here's a tentative example to illustrate my idea. (This is not a fully
thought-up example/syntax; it's just a way of trying to explain the
concept):
Java source (j2se)
-------------------
...
// @Processor(add to j2me
//
// float some_field;
//
// )
@Processor(Not in j2me!)
int some_field =
some +
initializing()
code;
...
Processed source (target => j2me)
----------------------------------
...
@BeginRevert(add to j2me)
float some_field;
@EndRevert()
// @Revert(@Processor(Not in j2me!)
// int some_field =
// some +
// initializing()
// code; )
...
Modified processed source ;-)
-----------------------------
...
@BeginRevert(add to j2me)
float some_field = missing initializer!;
@EndRevert()
// @Revert(@Processor(Not in j2me!)
// int some_field =
// some +
// initializing()
// code; )
...
Revert-processed source (=> ready for "svn ci")
-----------------------------------------------
...
// @Processor(add to j2me)
//
// float some_field = missing initializer!;
//
// )
@Processor(Not in j2me!)
int some_field =
some +
initializing()
code;
...
The "intelligent" part comes with the idea that the @Processor
directive, in the j2se uncommented source above, "knows" that it applies
to the whole field declaration, including any initializer, regardless of
the number of lines used by the field declaration. Such directives
could be "limited" to work on:
1- classes/interface declarations
2- method/field declarations
3- static statement blocks
4- statement blocks in general
5- other things?
The idea is that it is not a "character/lined based" processor, but
really a syntax-driven one.
Using comments (as above) is optional, yet I recommend doing it so that
text editors don't syntax color unapplied code in any current situation
(source, processed-source).
Both j2se & j2me (as well as other target developers) will be able to
work on the "processed" and the original source code. Typically, the
revert processor will be used by j2me developers before "svn ci" of
their modifications.
It would be quite easy for me to develop such a tool using SableCC.
FYI, the output of SableCC is not encumbered by any copyright other than
whatever the "grammar" developer wants. Also, I will contribute the
developed tool to Harmony if this approach is chosen (or develop it as a
Harmony developer, if you prefer...).
[Geir: Yes, I know, I have to send the paperwork...]
Some advantages:
1- Even if you package "processed source" into a jar file (for j2me
users), line numbers match exactly. Bug reports from users won't be
about Mars or Jupiter. :-)
2- Debugging using both initial and processed source works flawlessly.
3- Major code reorganization can be done cleanly without the need for
bug-prone, tricky, hand-made merges (as automated merge can be
misleading in such case)
4- We can make the tool and its syntax as "gentle" as you want; we have
total control on its syntax and semantics. :-)
5- We could add a "syntax checking" mode to the processor, so that in
that mode, *all* code would be checked for syntax correctness (even
commented processor directives) instead of modified for a specific
target. This could be added to the ant script for all developers, so
that at no point somebody "svn ci" syntactically invalid code in case
they, by error, changed some of the commented directives...
6- We could even implement a "not in processed form" check mode that
could be added as a pre-commit check phase, so that we make sure code is
never checked-in in processed form.
Anyway, that's an idea you could play with.
Have fun!
Etienne
Geir Magnusson Jr. wrote:
> So we all agree that it's not an ideal solution.
>
> Can anyone think of anything else? No one said this was going to be
> easy...
--
Etienne M. Gagnon, Ph.D. http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/