Re: Proposed SVN Reorg

[John Fallows <jo...@gmail.com>]

On 5/31/05, Martin Marinschek <ma...@gmail.com> wrote:
> Thanks - looks great, share issues are resolved with that proposal.
> 
> John?

Sorry for the late response, have been very busy lately. :-)

See responses inline below.

> On 5/30/05, Sean Schofield <se...@gmail.com> wrote:
> > Here is a first pass at a SVN reorg plan.  The idea is to reorg svn
> > into subprojects making it easier for those interested in specific
> > subprojects to find the relevant code.  Struts recently reorganized
> > along simliar lines and I find it to be very useful.  For instance,
> > right now I'm not interested in struts core so I can use svn to
> > checkout just the shale stuff.
> >
> > current***
> > build
> > conf
> > core
> >    api
> >    api/src
> >    api/src/java
> >    api/src/test
> >    build*
> >    impl
> >    impl/src
> >    impl/src/java
> >    impl/src/test
> >    share
> >    share/src
> >    share/src/java
> >    share/src/test
> > lib
> > examples
> >    build*
> >    src/java
> >    src/test
> > sandbox
> >    build*
> >    src
> >    src/java
> >    src/test
> > site
> > tomahawk
> >    build*
> >    src/java
> >    src/test
> >    share/src
> >    share/src/java
> >    share/src/test
> > xdocs
> >
> > * = svn:external link to the build subrproject
> > ** = svn:external link to the shared source code (original maintained
> > inside core subrpoject)
> > ***= svn:external link to the trunk of all subprojects
> >
> > The build directory would also be available in each project even
> > though there is really only one master copy.  The idea is that if you
> > checked out only the sandbox project
> > (http://svn.apache.org/respos/asf/myfaces/sandbox) you would have a
> > build directory with the same build.xml used by all subprojects (and
> > the project as a whole.)  So you could perform the builds with no
> > problem.  I still need to give some thought as to how we could pull
> > this off without also sharing the dependencies in lib (I have a
> > feeling this could be where Maven comes into play.)

You are right about Maven having an impact here.  In fact, with Maven
in the picture, there would be no need for svn:externals to deliver a
managed duplication of build code across projects.  Instead, common
tasks would be factored into Maven plugins, fostering reuse.  Maven
will also manage the dependencies automatically, as you indicated.

Btw, one of the focus areas for Maven2 is making it really simple to
write plugins, so I'd recommend taking a look at M2 at the same time
you investigate M1.

> > The core subproject would consist of api + impl stuff.  I thought
> > about calling this implementation but I didn't want to confuse that
> > with the impl.jar code.  Struts has a subproject called core so I
> > thought I would suggest it.  The core subproject also contains the
> > "share" code used by tomahawk.  I am proposing a svn:external link in
> > the tomahawk project so that when you check out tomahawk
> > (http://svn.apache.org/repos/asf/myfaces/tomahawk) you automatically
> > get a directory called share with the shared code in it.  BTW, commits
> > against this "linked" directory are commited to the actual directory
> > and updates against the link work against the real directory.  I think
> > this is a nice way to share the code but still allow tomahawk to be
> > checked out as a complete project and compiled (without having to
> > checkout the core stuff.)

This solution nicely solves the management of the share source code
across projects.  However, it does not appear to address the runtime
aspect.  Given that "core" and "tomahawk" will be released
independently (right?), then upgrading to a new version of "tomahawk"
would implicitly change the code used by the "core" runtime, even
though that version has not been upgraded.  The implicit WEB-INF/lib
JAR ordering in the classpath might cause either instance of the share
package to be eclipsed, but we do know that one will definitely
eclipse the other.

I believe there can be no shared packages across "core" and "tomahawk"
that are not explicitly called out as a versioned project, with a
public API commitment across releases.  Since we've been trying very
hard to avoid creating an independent project, and we don't want to
pollute the MyFaces API JAR, we might consider repackaging the share
code into a tomahawk-specific Java package automatically during build
to avoid the eclipsing problems described above.

The "tomahawk" project should also be split into "api" and "impl"
otherwise it is all effectively "api".

> > Most of the directories would also have a trunk, branches and tags
> > folders (not shown in the list for simplicity's sake.)  Take a look at
> > how this works in struts.  If you wanted to check out the latest
> > version of the core you would use
> > http://svn.apache.org/repos/asf/myfaces/core/trunk.
> >
> > James Mitchell and I are hoping to experiment with a backup copy of
> > the repository before anything is attempted.  Also of course we will
> > need to get PMC approval for this kind of a change.  If this is close
> > to what we want, it might be possible to setup an external svn server
> > for people to evaluate the new scheme.  I'd rather have agreement on
> > 90% of this though before doing all of the work to move things around.
> >
> > Ultimately this won't change things much for developers who use the
> > entire myfaces project.  You would checkout myfaces/current and get
> > everything.  The build file would point to new directories, etc. and
> > would generate the jars that we talked about on the earlier thread.
> > Of course there would probably be some tweaks to your IDE project
> > file.

I think we should revisit the need for "current" after assessing the
dependency management facilities in Maven / Maven2.

Kind Regards,
John Fallows.

Fwd: Proposed SVN Reorg

[Martin Marinschek <ma...@gmail.com>]

---------- Forwarded message ----------
From: Martin Marinschek <ma...@gmail.com>
Date: Jun 9, 2005 9:04 AM
Subject: Re: Proposed SVN Reorg
To: John Fallows <jo...@gmail.com>


Sounds tedious for developing, though...

If I do a change in the share package, I will need to run this
mechanism to generate the share packages from that.

Plus: we really have two codebases then.

I'd rather have a possibility where we just program against the common
share package, and then, upon build, the share-code is moved to a new
package and the tomahawk code is refactored to use this package.

I haven't got a clue though if this is possible with the available build tools!

regards,

Martin

On 6/9/05, John Fallows <jo...@gmail.com> wrote:
> > You are right, the runtime problem with share persists...
>
> > I assume you are proposing something like Sun does in its JWSDP -
> > packaging Xerces along and refactoring it to com.sun.* classes?
>
> This is one possibility, other suggestions are also welcome. :-)
>
> > And you are only proposing it for the deployed jar-files, not for development?
>
> We would need to consider the impact of all code in other parts of the
> Tomahawk implementation that reference the shared codebase.  Would it
> use the Core package or the Tomahawk package?
>
> This code sharing technique is more self-contained if the Tomahawk
> codebase only references the Tomahawk packages.  Therefore, we might
> be better off treating the shared codebase as a set of templates that
> are used to generate code for the appropriate package in each of the
> Core and Tomahawk projects.
>
> Kind Regards,
> John Fallows.
>
>

Re: Proposed SVN Reorg

[Martin Marinschek <ma...@gmail.com>]

Hi Sean,

welcome back!

I think we can definitely stick with your original proposal, even
though the discussion has not ended between John and me. (Blame me!)

What John wants is something he would need to propose seperately from
the current svn reorg, and which will need to be discussed separately.

regards,

Martin

On 6/19/05, Sean Schofield <se...@gmail.com> wrote:
> I wanted to jump in here and see if we can't resolve the problem of
> the shared (or core) code so we can get started on the reorg.
> 
> I'm definitely -1 on having a separate core.jar/share.jar file.  I'm
> not sure if anyone is proposing that but I'm against that idea.  We
> don't have that now and IMO this would add to the confusion.  Craig
> has been suggesting we continue to maintain the two jar approach: one
> for api, one for impl.  I agree we should stick with that pattern.  We
> will also have the myfaces-all.jar as a matter of convenience to the
> users.
> 
> Recall the easrlier discussion where we talked about the "artifacts"
> of the process (http://wiki.apache.org/myfaces/MyFaces_Artifacts).
> Its hard to imagine a scenario where we would release a new version of
> tomahawk but now release a new version of myfaces (because of the
> shared code.)  Those using the RI with Tomahawk, however, need only to
> download the new tomahawk.jar.
> 
> As for Maven and svn:externals I would say we still definitely need
> svn:externals even if we ultimately move to Maven.  I feel pretty
> strongly that everything needs to be buildable using Ant since that is
> the industry standard right now (not Maven.)  We can of course builld
> our releases, run tests, publish the website, etc. using Maven but the
> average user needs to be able to checkout myfaces/current and get
> everything.
> 
> I'd like to get started on the svn reorg ASAP.  Are there any major
> issues with the original proposal I outlined?  I can't think of a
> change to the way we handle the share/core code that would be an
> improvement over what we have already discussed.
> 
> sean
> 
> On 6/9/05, John Fallows <jo...@gmail.com> wrote:
> > > I didn't mean the act of the generation, I know this is but executing
> > > a command - but if I need to develop on the implementation or the
> > > components source code and have to execute the command whenever I
> > > change something in the share classes, I promise this makes developing
> > > much more tedious, and this is probably not what we should encourage.
> >
> > We make heavy use of code generation in ADF Faces, and do not find it
> > to be a huge issue in practice.  That might well be due to the fact
> > that, in practice, the source of those generated files have a low
> > frequency of update.
> >
> > When such an update is necessary, and we run the command line build
> > tool, the generated files are still part of the IDE experience, so the
> > overall development impact is slight.
> >
> > > Rather than imposing this on the developers, I would impose on the
> > > users to either download and use myfaces-all.jar if they use
> > > components and implementation or myfaces-components.jar if they just
> > > use the components (and implicitly package share along with that).
> > >
> > > Not so much more inconvenience for the users, a lot more convenient
> > > for us developers!
> >
> > I might be missing something, but this proposal doesn't seem to
> > address the runtime eclipsing of shared classes, which looks to still
> > be an issue because core and extensions are released independently.
> >
> > Given the choice of whether to impose upon developers who understand
> > the implications of their decisions or upon users who often do not
> > understand the subtleties, IMHO, I would suggest the former, since
> > developers are better equipped to deal with it.
> >
> > Kind Regards,
> > John Fallows
> >
>

Re: Proposed SVN Reorg

[Manfred Geiler <ma...@gmail.com>]

> Recall the easrlier discussion where we talked about the "artifacts"
> of the process (http://wiki.apache.org/myfaces/MyFaces_Artifacts).
> Its hard to imagine a scenario where we would release a new version of
> tomahawk but now release a new version of myfaces (because of the
> shared code.)  Those using the RI with Tomahawk, however, need only to
> download the new tomahawk.jar.

Yes, that's also my feeling.
Both parts (impl and tomahawk) should be released concurrently. At
least for major releases.
If tomahawk (for what reason ever) needs to be released separately it
is the task of the release technician(s) to make sure that there are
no incompatibilities to the current myfaces implementation (i.e. there
where no non-trivial changes in the shared code since the last impl
release). So, one step during the tomahawk release vote would be to
test the tomahawk components against the last official myfaces impl.
If there are problems or incompatibilities, the decision (release or
not) is made by voting. Two possible results:
1. Release is done with a "disclaimer" in the release notes
2. Release of tomahawk is deferred until the next official myfaces impl

I think this is a pragmatical solution taking into acount, that the
shared classes are mainly common base classes that won't change every
day.

-Manfred

Re: Proposed SVN Reorg

[Sean Schofield <se...@gmail.com>]

I wanted to jump in here and see if we can't resolve the problem of
the shared (or core) code so we can get started on the reorg.

I'm definitely -1 on having a separate core.jar/share.jar file.  I'm
not sure if anyone is proposing that but I'm against that idea.  We
don't have that now and IMO this would add to the confusion.  Craig
has been suggesting we continue to maintain the two jar approach: one
for api, one for impl.  I agree we should stick with that pattern.  We
will also have the myfaces-all.jar as a matter of convenience to the
users.

Recall the easrlier discussion where we talked about the "artifacts"
of the process (http://wiki.apache.org/myfaces/MyFaces_Artifacts). 
Its hard to imagine a scenario where we would release a new version of
tomahawk but now release a new version of myfaces (because of the
shared code.)  Those using the RI with Tomahawk, however, need only to
download the new tomahawk.jar.

As for Maven and svn:externals I would say we still definitely need
svn:externals even if we ultimately move to Maven.  I feel pretty
strongly that everything needs to be buildable using Ant since that is
the industry standard right now (not Maven.)  We can of course builld
our releases, run tests, publish the website, etc. using Maven but the
average user needs to be able to checkout myfaces/current and get
everything.

I'd like to get started on the svn reorg ASAP.  Are there any major
issues with the original proposal I outlined?  I can't think of a
change to the way we handle the share/core code that would be an
improvement over what we have already discussed.

sean

On 6/9/05, John Fallows <jo...@gmail.com> wrote:
> > I didn't mean the act of the generation, I know this is but executing
> > a command - but if I need to develop on the implementation or the
> > components source code and have to execute the command whenever I
> > change something in the share classes, I promise this makes developing
> > much more tedious, and this is probably not what we should encourage.
> 
> We make heavy use of code generation in ADF Faces, and do not find it
> to be a huge issue in practice.  That might well be due to the fact
> that, in practice, the source of those generated files have a low
> frequency of update.
> 
> When such an update is necessary, and we run the command line build
> tool, the generated files are still part of the IDE experience, so the
> overall development impact is slight.
> 
> > Rather than imposing this on the developers, I would impose on the
> > users to either download and use myfaces-all.jar if they use
> > components and implementation or myfaces-components.jar if they just
> > use the components (and implicitly package share along with that).
> >
> > Not so much more inconvenience for the users, a lot more convenient
> > for us developers!
> 
> I might be missing something, but this proposal doesn't seem to
> address the runtime eclipsing of shared classes, which looks to still
> be an issue because core and extensions are released independently.
> 
> Given the choice of whether to impose upon developers who understand
> the implications of their decisions or upon users who often do not
> understand the subtleties, IMHO, I would suggest the former, since
> developers are better equipped to deal with it.
> 
> Kind Regards,
> John Fallows
>

Re: Proposed SVN Reorg

[John Fallows <jo...@gmail.com>]

> I didn't mean the act of the generation, I know this is but executing
> a command - but if I need to develop on the implementation or the
> components source code and have to execute the command whenever I
> change something in the share classes, I promise this makes developing
> much more tedious, and this is probably not what we should encourage.

We make heavy use of code generation in ADF Faces, and do not find it
to be a huge issue in practice.  That might well be due to the fact
that, in practice, the source of those generated files have a low
frequency of update.

When such an update is necessary, and we run the command line build
tool, the generated files are still part of the IDE experience, so the
overall development impact is slight.

> Rather than imposing this on the developers, I would impose on the
> users to either download and use myfaces-all.jar if they use
> components and implementation or myfaces-components.jar if they just
> use the components (and implicitly package share along with that).
> 
> Not so much more inconvenience for the users, a lot more convenient
> for us developers!

I might be missing something, but this proposal doesn't seem to
address the runtime eclipsing of shared classes, which looks to still
be an issue because core and extensions are released independently.

Given the choice of whether to impose upon developers who understand
the implications of their decisions or upon users who often do not
understand the subtleties, IMHO, I would suggest the former, since
developers are better equipped to deal with it.

Kind Regards,
John Fallows

Fwd: Proposed SVN Reorg

[Martin Marinschek <ma...@gmail.com>]

I didn't mean the act of the generation, I know this is but executing
a command - but if I need to develop on the implementation or the
components source code and have to execute the command whenever I
change something in the share classes, I promise this makes developing
much more tedious, and this is probably not what we should encourage.

Rather than imposing this on the developers, I would impose on the
users to either download and use myfaces-all.jar if they use
components and implementation or myfaces-components.jar if they just
use the components (and implicitly package share along with that).

Not so much more inconvenience for the users, a lot more convenient
for us developers!

regards,

Martin

---------- Forwarded message ----------
From: John Fallows <jo...@gmail.com>
Date: Jun 9, 2005 4:38 PM
Subject: Re: Proposed SVN Reorg
To: martin@marinschek.com


> Sounds tedious for developing, though...

It is approximately as tedious as any other kind of Java source generation. :-)

> If I do a change in the share package, I will need to run this
> mechanism to generate the share packages from that.

Right.  Is that a frequent occurrence?

> Plus: we really have two codebases then.

How so?  We generate two packages to avoid eclipsing, but we still
have only one source of truth in the templates, right?

> I'd rather have a possibility where we just program against the common
> share package, and then, upon build, the share-code is moved to a new
> package and the tomahawk code is refactored to use this package.

The danger here is that we would be doing daily testing (during
development) of different code than we ship.

If we develop both Core and Tomahawk in the same IDE Workspace,
wouldn't we still suffer from the class eclipsing problem?

> I haven't got a clue though if this is possible with the available build tools!

Attempting to take this approach will make the Maven (or any build
tool) integration difficult.

Each Java source from Tomahawk that references a shared class in the
Core package would have to be modified to reference that shared class
in the repackaged Tomahawk location.

However, it would be bad practice to modify those referencing sources
in-place, because they would then look like local modifications to be
committed to the source repository.  Therefore, the modified
referencing sources would need to be generated to a different location
(such as "target/src/java".

Unfortunately, Javac gets rather upset when a class is defined by
multiple source files in the same source path, and fails compilation.
But, the source path is defined by just the top-level source
directories, such as "src/java" and "target/src/java".  I think Javac
automatically picks up any .java files under each source path root,
but it might be possible to provide an exclusion list.

So, either we'd have to use a different extension for files that
reference the shared codebase (which will lead to poor IDE
experience), or we need to track the exclusion list of files that
reference the shared code base.

IMHO, either of these would be more tedious than remembering to run
codegen each time we modify the shared source templates.

Kind Regards,
John Fallows.

Re: Proposed SVN Reorg

[John Fallows <jo...@gmail.com>]

> You are right, the runtime problem with share persists...

> I assume you are proposing something like Sun does in its JWSDP -
> packaging Xerces along and refactoring it to com.sun.* classes?

This is one possibility, other suggestions are also welcome. :-)

> And you are only proposing it for the deployed jar-files, not for development?

We would need to consider the impact of all code in other parts of the
Tomahawk implementation that reference the shared codebase.  Would it
use the Core package or the Tomahawk package?

This code sharing technique is more self-contained if the Tomahawk
codebase only references the Tomahawk packages.  Therefore, we might
be better off treating the shared codebase as a set of templates that
are used to generate code for the appropriate package in each of the
Core and Tomahawk projects.

Kind Regards,
John Fallows.

Re: Proposed SVN Reorg

[Martin Marinschek <ma...@gmail.com>]

You are right, the runtime problem with share persists...

I assume you are proposing something like Sun does in its JWSDP -
packaging Xerces along and refactoring it to com.sun.* classes?

And you are only proposing it for the deployed jar-files, not for development?

That would be a workable solution, I would say.

regards,

Martin

On 6/8/05, John Fallows <jo...@gmail.com> wrote:
> On 5/31/05, Martin Marinschek <ma...@gmail.com> wrote:
> > Thanks - looks great, share issues are resolved with that proposal.
> >
> > John?
> 
> Sorry for the late response, have been very busy lately. :-)
> 
> See responses inline below.
> 
> > On 5/30/05, Sean Schofield <se...@gmail.com> wrote:
> > > Here is a first pass at a SVN reorg plan.  The idea is to reorg svn
> > > into subprojects making it easier for those interested in specific
> > > subprojects to find the relevant code.  Struts recently reorganized
> > > along simliar lines and I find it to be very useful.  For instance,
> > > right now I'm not interested in struts core so I can use svn to
> > > checkout just the shale stuff.
> > >
> > > current***
> > > build
> > > conf
> > > core
> > >    api
> > >    api/src
> > >    api/src/java
> > >    api/src/test
> > >    build*
> > >    impl
> > >    impl/src
> > >    impl/src/java
> > >    impl/src/test
> > >    share
> > >    share/src
> > >    share/src/java
> > >    share/src/test
> > > lib
> > > examples
> > >    build*
> > >    src/java
> > >    src/test
> > > sandbox
> > >    build*
> > >    src
> > >    src/java
> > >    src/test
> > > site
> > > tomahawk
> > >    build*
> > >    src/java
> > >    src/test
> > >    share/src
> > >    share/src/java
> > >    share/src/test
> > > xdocs
> > >
> > > * = svn:external link to the build subrproject
> > > ** = svn:external link to the shared source code (original maintained
> > > inside core subrpoject)
> > > ***= svn:external link to the trunk of all subprojects
> > >
> > > The build directory would also be available in each project even
> > > though there is really only one master copy.  The idea is that if you
> > > checked out only the sandbox project
> > > (http://svn.apache.org/respos/asf/myfaces/sandbox) you would have a
> > > build directory with the same build.xml used by all subprojects (and
> > > the project as a whole.)  So you could perform the builds with no
> > > problem.  I still need to give some thought as to how we could pull
> > > this off without also sharing the dependencies in lib (I have a
> > > feeling this could be where Maven comes into play.)
> 
> You are right about Maven having an impact here.  In fact, with Maven
> in the picture, there would be no need for svn:externals to deliver a
> managed duplication of build code across projects.  Instead, common
> tasks would be factored into Maven plugins, fostering reuse.  Maven
> will also manage the dependencies automatically, as you indicated.
> 
> Btw, one of the focus areas for Maven2 is making it really simple to
> write plugins, so I'd recommend taking a look at M2 at the same time
> you investigate M1.
> 
> > > The core subproject would consist of api + impl stuff.  I thought
> > > about calling this implementation but I didn't want to confuse that
> > > with the impl.jar code.  Struts has a subproject called core so I
> > > thought I would suggest it.  The core subproject also contains the
> > > "share" code used by tomahawk.  I am proposing a svn:external link in
> > > the tomahawk project so that when you check out tomahawk
> > > (http://svn.apache.org/repos/asf/myfaces/tomahawk) you automatically
> > > get a directory called share with the shared code in it.  BTW, commits
> > > against this "linked" directory are commited to the actual directory
> > > and updates against the link work against the real directory.  I think
> > > this is a nice way to share the code but still allow tomahawk to be
> > > checked out as a complete project and compiled (without having to
> > > checkout the core stuff.)
> 
> This solution nicely solves the management of the share source code
> across projects.  However, it does not appear to address the runtime
> aspect.  Given that "core" and "tomahawk" will be released
> independently (right?), then upgrading to a new version of "tomahawk"
> would implicitly change the code used by the "core" runtime, even
> though that version has not been upgraded.  The implicit WEB-INF/lib
> JAR ordering in the classpath might cause either instance of the share
> package to be eclipsed, but we do know that one will definitely
> eclipse the other.
> 
> I believe there can be no shared packages across "core" and "tomahawk"
> that are not explicitly called out as a versioned project, with a
> public API commitment across releases.  Since we've been trying very
> hard to avoid creating an independent project, and we don't want to
> pollute the MyFaces API JAR, we might consider repackaging the share
> code into a tomahawk-specific Java package automatically during build
> to avoid the eclipsing problems described above.
> 
> The "tomahawk" project should also be split into "api" and "impl"
> otherwise it is all effectively "api".
> 
> > > Most of the directories would also have a trunk, branches and tags
> > > folders (not shown in the list for simplicity's sake.)  Take a look at
> > > how this works in struts.  If you wanted to check out the latest
> > > version of the core you would use
> > > http://svn.apache.org/repos/asf/myfaces/core/trunk.
> > >
> > > James Mitchell and I are hoping to experiment with a backup copy of
> > > the repository before anything is attempted.  Also of course we will
> > > need to get PMC approval for this kind of a change.  If this is close
> > > to what we want, it might be possible to setup an external svn server
> > > for people to evaluate the new scheme.  I'd rather have agreement on
> > > 90% of this though before doing all of the work to move things around.
> > >
> > > Ultimately this won't change things much for developers who use the
> > > entire myfaces project.  You would checkout myfaces/current and get
> > > everything.  The build file would point to new directories, etc. and
> > > would generate the jars that we talked about on the earlier thread.
> > > Of course there would probably be some tweaks to your IDE project
> > > file.
> 
> I think we should revisit the need for "current" after assessing the
> dependency management facilities in Maven / Maven2.
> 
> Kind Regards,
> John Fallows.
> 
>