Re: [classlib] Preprocessor - an attempt at [even more] clarity

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> Tim Ellison wrote:
>>> There are multiple 'devtarget's -- each contains all the source code
>>> marked-up for every target being developed, but each is distinct by
>>> having different targets uncommented.  Since each devtarget contains the
>>> entire source code it is possible to accurately transform from any one
>>> devtarget to any other devtarget.  One of the devtargets is actually
>>> stored in svn, and that is the canonical form of the devtarget code.
>> So that's 'master'. and everythign else is a transformed master, right?
> 
> Yes, but some transforms preserve all mark-up and are used to switch the
> compilable form between different targets (these are the 'devtarget's);
> and some transforms are mark-up-lossy and are used to provide the end
> user compilable form (this is the 'releasetarget').

Yes, I figured that out last night when I was driving to dinner. Let me 
  see if I have it right :

       (Master)
      /   |    \
     /    |     \
    /     |      \
dev1 <-> dev2 <->dev3
  |       |        |
targ1   targ2   targ3

where in your case, master doesn't exist (consider it virtual), dev1/2/3 
are compilable under whatever platform they are targeted to yet contain 
all the stuff for other platforms, and targ1/2/3 are 'clean code' code 
the target platform.

There exists a transform process

    interDev(dev1, 2) -> dev2

which is really conceptually

    masterToDev(devToMaster(dev1), 2)

(if you had a master) and further

    toCleanTarget(dev1) - > targ1

('normal' code for target platform)



> 
>>> Screech -- the releasetarget, by definition, does not contain the source
>>> for all possible targets so that it is consumable by end-users.
>> That is correct.
>>
>> The only code that contains all code is master.
> 
> Not in the technique that I'm talking about.  All devtargets contain all
> the code (but they differ in which code is compilable and which is in
> comments).

Right - because you don't have a master

> 
> Let me try using very simple example for my favourite type, assuming we
> were doing CLDC and SE1.5 development simultaneously. 

[SNIP]

Thanks for the very nice example.  That example conforms w/ my picture 
of how things might work (although I suspect things could get *very* 
ugly to work with in the intermediate dev1/2/3 form, but I think that 
some real use cases will tell us that)

> 
> 
> We then talked about how the different devtargets can be stored in
> different working directories for use by Harmony developers with
> file-based tools, or (thinking about it for a second now) it could be an
> on the fly processing to/from canonical form on disk by the IDE editor.

Yes - that latter part is exactly what I've been advocating :

One copy (the master) on which the IDE and cmd line tooling does both 
the "file/directory" masking (dealing with the differences that aren't 
just in class files - add and remove classes, etc - the "lookaside 
table") as well as code xform.

I never thought of having that intermediate form (dev1/2/3) as being 
fundamental to the model, although it's a nice option. I can see how it 
will be nice for editing, but I'm instinctively suspicious of it being 
necessary).

Why?

Because I can imagine that as a SE programmer, I would want 1 of 2 things :

1) clean code to work on for a specific target (say SE 5)

2) a hybrid version of your dev1 that only has the glop for a subset of 
the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus 
across just those two.

Maybe it's just me, but I'd probably want the distractions of other 
platforms out of the code so I can get better into the 'flow' when 
developing.

So that's why I think of dev1/2/3 as being the virtual code (not key to 
the process) that can be generated either by the tooling onto disk as a 
temp form, or by IDE virtually in situ.

Further, there aren't just N dev forms for the N distinct 
platforms/version, but rather the freedom for combinations where they 
make sense (a dev w/ SE 6 and SE 6 only)

If you want to generate a dev1 and work on that on disk, great - do it, 
you can go back to master.

If you want to just work on the single master tree, you can do that if 
the IDE has the tooling.

> 
> We also talked about how patches to the source that Mrs CLDC developer
> sees can be matched back to a devtarget.

Yes, I did read it :)

Thanks for the example.  I guess the difference is that I virtualized 
the dev1/2/3 form and used tooling to just let people work on master 
directly, or let dev1/2/3 be concrete as a temporary form for 
convenience, as I noted above.

Or in simpler terms, the model differences seem to boil down to this. 
Given N possible targets:

1) Master is concrete, dev1..M (where M possibly > N) are generated 
simply for ease of use either as concrete tree on disk or virtually in IDE

2) Master is virtual, dev1..N are concrete, one dev form is Most Favored 
Nation Status so something coherent can be shoved into SVN (Q: is MFNS 
necessary? Nice, yes, but necessary?)

geir

Re: [classlib] Preprocessor - an attempt at [even more] clarity

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>>
>> Because I can imagine that as a SE programmer, I would want 1 of 2 things :
>>
>> 1) clean code to work on for a specific target (say SE 5)
>>
>> 2) a hybrid version of your dev1 that only has the glop for a subset of
>> the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus
>> across just those two.
>>
>> Maybe it's just me, but I'd probably want the distractions of other
>> platforms out of the code so I can get better into the 'flow' when
>> developing.
>>
>> So that's why I think of dev1/2/3 as being the virtual code (not key to
>> the process) that can be generated either by the tooling onto disk as a
>> temp form, or by IDE virtually in situ.
> 
> but how do you resolve the ambiguity of where new/moved code should go
> in the master if you are working in a dev that has lost the context of
> other streams?

I want to make sure I understand what you are asking - I think you are 
asking the question "As a developer, if I can't see the content of the 
other streams, how can I place the changes in context of the other streams?"

It's not clear to me that there's a big problem here - the fact that a 
complete master exists means that when you "commit" your changes to the 
master, you can resolve ambiguities in placement relative to other 
streams at that time, *if* there are any.  (For the life of me, I can't 
figure out how to mechanically characterize an ambiguity...)

The big question here is if that's a reasonable approach, and the answer 
will be based on how often it happens.

If it happens in 1 in 50 "commits", I'd argue that the benefit of the 
reduction of clutter is worth it.  If it happens every time, it's 
clearly going to be a royal PITA.  I don't know actually where to draw 
that line.  I expect testing with toy examples will help us figure it out.

> 
> it's the point that was made here:
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200611.mbox/%3c454A4185.6070700@sablevm.org%3e
> 

I went back and review that, and I keep getting this nagging feeling 
that we're being a bit naive about this, but I can't put my finger on it 
and explain why :)  I'll try to explain in a separate post.

> it would be cool if we could do that.

Yes - my goal here is to make this as transparent as possible to the 
working harmony developer.

> 
>> Further, there aren't just N dev forms for the N distinct
>> platforms/version, but rather the freedom for combinations where they
>> make sense (a dev w/ SE 6 and SE 6 only)
>>
>> If you want to generate a dev1 and work on that on disk, great - do it,
>> you can go back to master.
> 
> Only if the reverse mapping is exact, which I believe means that you
> need to take the full master context along with you.

I don't believe so. I need to do some tests.  The reason I don't believe 
so simply because the master context exists in the master (by defn), 
which always is a part of the "commit" process - IOW, when committing a 
  dev1' (modified dev1), you can regenerate the clean original dev1 from 
master, then find your diffs to that from dev1' (the modified thing you 
are committing), and map those back into the master.

I think.

[SNIP]

>>
>> Or in simpler terms, the model differences seem to boil down to this.
>> Given N possible targets:
>>
>> 1) Master is concrete, dev1..M (where M possibly > N) are generated
>> simply for ease of use either as concrete tree on disk or virtually in IDE
>>
>> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
>> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
>> necessary? Nice, yes, but necessary?)
> 
> Yes, that seems to be the difference.  I'd like to be convinced that
> working in dev1..M code is possible in approach (1).

Yes, and I can almost prove now that not having a master is unworkable, 
because we're not talking only about file content, but different files, 
and different directory structure.  So if the classlib code spans both 
versions (SE 5, SE 6) as well as platforms (ME, SE), the build process 
will require processing to deal with that during the build.  if that's 
the case, then you are in case 1), not 2).

I'm going to try to cough up some simple use-cases of code trees and 
toss in the sandbox.

geir

Re: [classlib] Preprocessor - an attempt at [even more] clarity

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> One copy (the master) on which the IDE and cmd line tooling does both
> the "file/directory" masking (dealing with the differences that aren't
> just in class files - add and remove classes, etc - the "lookaside
> table") as well as code xform.
> 
> I never thought of having that intermediate form (dev1/2/3) as being
> fundamental to the model, although it's a nice option. I can see how it
> will be nice for editing, but I'm instinctively suspicious of it being
> necessary).
> 
> Why?
> 
> Because I can imagine that as a SE programmer, I would want 1 of 2 things :
> 
> 1) clean code to work on for a specific target (say SE 5)
> 
> 2) a hybrid version of your dev1 that only has the glop for a subset of
> the platforms (say just Java SE 5 and Java SE 6) in it, so I can focus
> across just those two.
> 
> Maybe it's just me, but I'd probably want the distractions of other
> platforms out of the code so I can get better into the 'flow' when
> developing.
> 
> So that's why I think of dev1/2/3 as being the virtual code (not key to
> the process) that can be generated either by the tooling onto disk as a
> temp form, or by IDE virtually in situ.

but how do you resolve the ambiguity of where new/moved code should go
in the master if you are working in a dev that has lost the context of
other streams?

it's the point that was made here:
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200611.mbox/%3c454A4185.6070700@sablevm.org%3e

it would be cool if we could do that.

> Further, there aren't just N dev forms for the N distinct
> platforms/version, but rather the freedom for combinations where they
> make sense (a dev w/ SE 6 and SE 6 only)
> 
> If you want to generate a dev1 and work on that on disk, great - do it,
> you can go back to master.

Only if the reverse mapping is exact, which I believe means that you
need to take the full master context along with you.

> If you want to just work on the single master tree, you can do that if
> the IDE has the tooling.
> 
>>
>> We also talked about how patches to the source that Mrs CLDC developer
>> sees can be matched back to a devtarget.
> 
> Yes, I did read it :)

I was changing focus, you knew that <grr>

> Thanks for the example.  I guess the difference is that I virtualized
> the dev1/2/3 form and used tooling to just let people work on master
> directly, or let dev1/2/3 be concrete as a temporary form for
> convenience, as I noted above.
> 
> Or in simpler terms, the model differences seem to boil down to this.
> Given N possible targets:
> 
> 1) Master is concrete, dev1..M (where M possibly > N) are generated
> simply for ease of use either as concrete tree on disk or virtually in IDE
> 
> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
> necessary? Nice, yes, but necessary?)

Yes, that seems to be the difference.  I'd like to be convinced that
working in dev1..M code is possible in approach (1).

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Alexey Petrenko <al...@gmail.com>]

Some concreteness would be great :)

2006/11/3, Tim Ellison <t....@gmail.com>:
> Geir Magnusson Jr. wrote:
> > As fun as this discussion is, it's all theoretical.
>
> There's no harm in a bit of fun :-)
>
> So how about we create a sandbox in SVN and copy over a few types to
> mark-up while Etienne is developing the processor, and we can all watch
> that evolve?
>
> The sandbox ensures that we don't disturb people who are working in
> TRUNK (and therefore not having fun ;-p) and we should plan now to throw
> away the sandbox and do the mark-up again once we settle on a workable
> solution, i.e. we'd let the sandbox code get out of date.
>
> I'm inclined to say that we tackle the 1.5/1.6 differences as our
> example.  Marking-up a module like NIO would be easy enough.
>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

["Geir Magnusson Jr." <ge...@pobox.com>]

Etienne Gagnon wrote:
> Tim Ellison wrote:
>> I'll wait until you have a simple contrived example working and settled
>> upon some syntax for mark-up, then I'll help to mark-up some real
>> example and test the tooling.  Let me know when you have something ready
>> to try.
> 
> Yes,  I have to figure out some sane syntax for all this.  I'll try to
> find some easily extensible syntax, so that we don't have to break
> backward compatibility every time we extend it. :-)

it may help to keep that discussion here, so anyone with ideas can help...

geir

> 
> Etienne
> 

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Etienne Gagnon <eg...@sablevm.org>]

Tim Ellison wrote:
> I'll wait until you have a simple contrived example working and settled
> upon some syntax for mark-up, then I'll help to mark-up some real
> example and test the tooling.  Let me know when you have something ready
> to try.

Yes,  I have to figure out some sane syntax for all this.  I'll try to
find some easily extensible syntax, so that we don't have to break
backward compatibility every time we extend it. :-)

Etienne

-- 
Etienne M. Gagnon, Ph.D.            http://www.info2.uqam.ca/~egagnon/
SableVM:                                       http://www.sablevm.org/
SableCC:                                       http://www.sablecc.org/

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Tim Ellison <t....@gmail.com>]

Etienne Gagnon wrote:
> Tim Ellison wrote:
>>>> I'm inclined to say that we tackle the 1.5/1.6 differences as our
>>>> example.  Marking-up a module like NIO would be easy enough.
>>> That's what I was hoping for.  One concrete example :)
>>>
>>> I'm thrilled to see something happen in the sandbox (that's really the
>>> only way to do it... I'd be -1 to trying anything in the trunk.  I hope
>>> that you knew that...)
>> Yes, of course.  But I'll wait for Etienne to agree to the idea since
>> he's signing up to do the hard work.  I'll help to mark up the classlibs.
> 
> Of course, I agree!

I'll wait until you have a simple contrived example working and settled
upon some syntax for mark-up, then I'll help to mark-up some real
example and test the tooling.  Let me know when you have something ready
to try.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Etienne Gagnon <eg...@sablevm.org>]

Tim Ellison wrote:
>>>I'm inclined to say that we tackle the 1.5/1.6 differences as our
>>>example.  Marking-up a module like NIO would be easy enough.
>>
>>That's what I was hoping for.  One concrete example :)
>>
>>I'm thrilled to see something happen in the sandbox (that's really the
>>only way to do it... I'd be -1 to trying anything in the trunk.  I hope
>>that you knew that...)
> 
> Yes, of course.  But I'll wait for Etienne to agree to the idea since
> he's signing up to do the hard work.  I'll help to mark up the classlibs.

Of course, I agree!

Etienne

-- 
Etienne M. Gagnon, Ph.D.            http://www.info2.uqam.ca/~egagnon/
SableVM:                                       http://www.sablevm.org/
SableCC:                                       http://www.sablecc.org/

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> Tim Ellison wrote:
>> I'm inclined to say that we tackle the 1.5/1.6 differences as our
>> example.  Marking-up a module like NIO would be easy enough.
> 
> That's what I was hoping for.  One concrete example :)
> 
> I'm thrilled to see something happen in the sandbox (that's really the
> only way to do it... I'd be -1 to trying anything in the trunk.  I hope
> that you knew that...)

Yes, of course.  But I'll wait for Etienne to agree to the idea since
he's signing up to do the hard work.  I'll help to mark up the classlibs.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

["Geir Magnusson Jr." <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr. wrote:
>> As fun as this discussion is, it's all theoretical.
> 
> There's no harm in a bit of fun :-)
> 
> So how about we create a sandbox in SVN and copy over a few types to
> mark-up while Etienne is developing the processor, and we can all watch
> that evolve?
> 
> The sandbox ensures that we don't disturb people who are working in
> TRUNK (and therefore not having fun ;-p) and we should plan now to throw
> away the sandbox and do the mark-up again once we settle on a workable
> solution, i.e. we'd let the sandbox code get out of date.
> 
> I'm inclined to say that we tackle the 1.5/1.6 differences as our
> example.  Marking-up a module like NIO would be easy enough.

That's what I was hoping for.  One concrete example :)

I'm thrilled to see something happen in the sandbox (that's really the 
only way to do it... I'd be -1 to trying anything in the trunk.  I hope 
that you knew that...)

geir

Re: [classlib] Preprocessor - out of the abstractness came forth concreteness

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr. wrote:
> As fun as this discussion is, it's all theoretical.

There's no harm in a bit of fun :-)

So how about we create a sandbox in SVN and copy over a few types to
mark-up while Etienne is developing the processor, and we can all watch
that evolve?

The sandbox ensures that we don't disturb people who are working in
TRUNK (and therefore not having fun ;-p) and we should plan now to throw
away the sandbox and do the mark-up again once we settle on a workable
solution, i.e. we'd let the sandbox code get out of date.

I'm inclined to say that we tackle the 1.5/1.6 differences as our
example.  Marking-up a module like NIO would be easy enough.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [classlib] Preprocessor - an attempt at [even more] clarity

["Geir Magnusson Jr." <ge...@pobox.com>]

Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> [...]
>> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
>> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
>> necessary? Nice, yes, but necessary?)
> 
> I definitely think that it is necessary (where J2SE5 is the preferred
> devtarget), at least during the initial transition to using the
> processing tool.

:)

That still makes it "nice" - because you can always transform the SVN 
tree to any platform anyway.

> 
> It's (I think) the least intrusive method for starting to slowly add
> non-J2SE5 code in the development trunk while letting existing J2SE5
> developers continue developing without changing the way they've been
> using up to now.

Yep.  "nice".

> 
> By leaving the preferred devtarget in svn, the hordes of existing J2SE
> developers will have the choice of continuing to use simple "svn co/ci"
> to work on the code and won't have to learn to use new, yet not fully
> tested tools.

Yep.  "nice" :)

> 
> It also lets the smaller number of non-J2SE5 developers test the
> processing tools and help identify bugs (what!? :-P) and usability
> problems without forcing J2SE developers into doing the same.
> 
> Maybe, in the longer term, it would make sense to have some kind of
> canonical "master" form, and store that in svn, but I am a big fan of
> the least disturbance path, when trying to put in place new processes in
> an ongoing development project.  :-)
> 
> Question: What would be the advantage (in the longer term) of putting,
> in the Subversion trunk, source code in non-compilable canonical master
> form, instead of compilable preferred devtarget form?  Why is the
> existence of "master" so important to you?

I need to go check if where I said it wasn't to be compilable, as I 
don't believe I ever mandated that.  I do think it's going to be 
difficult to achieve though.

I think we both agree that we need a complete, fixed reference point to 
put in SVN. (In that sense, you actually have a master too, actually)

However, it's not clear to me that a master can actually be created that 
is complete for all platforms while still buildable directly w/o 
transforms and look-aside hiding/moving as part of the build process.

If you need to transform and move things around while building, then you 
don't have the model you are advocating.  You actually have the concrete 
master model.  You could cheat, and have both the concrete master tree 
and a preferred platform transform checked in at the same time, but 
that's just the concrete master model with a convenient "generated 
artifact" checkin as well.  That's fine, as long as you remember to 
update both and checkin both on modification.  We do that for the 
website, for example.


Even if we are astoundingly lucky, and we can create a simple buildable 
tree that can magically be transformed across platforms, I think that 
not distinguishing a master form as distinct from other forms is a 
mistake, because for the case of wanting a subset of representation for 
clarity in development (which I'm not sure you acknowledge yet is 
important), transforms are irreversible w/o a fixed master :

    masterToDev(master, 2+3, clean) -> dev2+3

meaning, transform master to contain code for platforms 2 and 3 only (ex 
Java SE 5 and 6), cleaning out all other, compilable for 2 but including 
3 material in comment form.

Then that is no longer a peer to your dev1/2/3 set (where all info is in 
all forms, just compilable on different platforms) but different because 
data for all platforms except 2 and 3 were taken out.

Therefore, I do think we need an explicit master copy, with the dev 
forms as intermediate convenience in whatever forms the developers 
choose to put them in (with all other stuff, subset of other stuff, w/o 
any other stuff...)

As fun as this discussion is, it's all theoretical.  I've never worked 
with multi-platform java code.  I've worked with a lot of multi-platform 
C code, and having the ability to ditch the confusing #ifdef cruft for 
the platforms I'm not working on at the moment would be a very 
innovative and pleasant advancement.  Anyway, this is theory.  The real 
test will be how it actually evolves in code...

geir

Re: [classlib] Preprocessor - an attempt at [even more] clarity

[Etienne Gagnon <eg...@sablevm.org>]

Geir Magnusson Jr. wrote:
> [...]
> 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored
> Nation Status so something coherent can be shoved into SVN (Q: is MFNS
> necessary? Nice, yes, but necessary?)

I definitely think that it is necessary (where J2SE5 is the preferred
devtarget), at least during the initial transition to using the
processing tool.

It's (I think) the least intrusive method for starting to slowly add
non-J2SE5 code in the development trunk while letting existing J2SE5
developers continue developing without changing the way they've been
using up to now.

By leaving the preferred devtarget in svn, the hordes of existing J2SE
developers will have the choice of continuing to use simple "svn co/ci"
to work on the code and won't have to learn to use new, yet not fully
tested tools.

It also lets the smaller number of non-J2SE5 developers test the
processing tools and help identify bugs (what!? :-P) and usability
problems without forcing J2SE developers into doing the same.

Maybe, in the longer term, it would make sense to have some kind of
canonical "master" form, and store that in svn, but I am a big fan of
the least disturbance path, when trying to put in place new processes in
an ongoing development project.  :-)

Question: What would be the advantage (in the longer term) of putting,
in the Subversion trunk, source code in non-compilable canonical master
form, instead of compilable preferred devtarget form?  Why is the
existence of "master" so important to you?

Etienne
PS: Tim, Thanks so much for providing clear answers and examples
explaining the approach!

-- 
Etienne M. Gagnon, Ph.D.            http://www.info2.uqam.ca/~egagnon/
SableVM:                                       http://www.sablevm.org/
SableCC:                                       http://www.sablecc.org/