Considering some re-org of SVN

[Marshall Schor <ms...@schor.com>]

I'm thinking about some re-org of our SVN layout based on these observations

-We're getting a lot of components.  Many of these are on somewhat 
different release cycles.

-We initially had a "main" node, a cpp node, a site note and a sandbox 
node.  The sandbox was for new-ish things.  As some of these things get 
more "mainline" - it would make sense to have them perhaps under another 
node to indicate that.  The idea would be that things in the sandbox 
were user-beware, but things in this other node were more "dependable" 
and "proven". 

Possible names for this other node might be: "parts".  Or we might want 
to have several names that categorized the kinds of things - such as 
"annotators", "servers", "embeddings", "corpora", "typeSystems", 
"tools", etc.

-The SVN conventions lean toward having branches and tags which are one 
level above the thing being released.  Right now, for sandbox projects, 
these are 2 levels above the released thing.  I think that, going 
forward, it would be better to go with the convention, following the 
convention-over-configuration philosophy, because the components are not 
likely to all be released on the same release cycle (although that would 
be a nice "goal" - like Eclipse does with it's many-part major releases).

Other opinions?

-Marshall

Re: Considering some re-org of SVN

[Thilo Goetz <tw...@gmx.de>]

Personally, I see no need for action, but I wouldn't
strongly oppose changes, either.

--Thilo

Marshall Schor wrote:
> I'm thinking about some re-org of our SVN layout based on these 
> observations
> 
> -We're getting a lot of components.  Many of these are on somewhat 
> different release cycles.
> 
> -We initially had a "main" node, a cpp node, a site note and a sandbox 
> node.  The sandbox was for new-ish things.  As some of these things get 
> more "mainline" - it would make sense to have them perhaps under another 
> node to indicate that.  The idea would be that things in the sandbox 
> were user-beware, but things in this other node were more "dependable" 
> and "proven".
> Possible names for this other node might be: "parts".  Or we might want 
> to have several names that categorized the kinds of things - such as 
> "annotators", "servers", "embeddings", "corpora", "typeSystems", 
> "tools", etc.
> 
> -The SVN conventions lean toward having branches and tags which are one 
> level above the thing being released.  Right now, for sandbox projects, 
> these are 2 levels above the released thing.  I think that, going 
> forward, it would be better to go with the convention, following the 
> convention-over-configuration philosophy, because the components are not 
> likely to all be released on the same release cycle (although that would 
> be a nice "goal" - like Eclipse does with it's many-part major releases).
> 
> Other opinions?
> 
> -Marshall
> 

Re: Considering some re-org of SVN

[Marshall Schor <ms...@schor.com>]

Adam Lally wrote:
> On Tue, Jul 8, 2008 at 12:21 PM, Marshall Schor <ms...@schor.com> wrote:
>   
>> Also, if I understand you right, you're suggesting that we have just an
>> "everything" package, instead of these "add-on" packages. If we have just an
>> everything package, would we not have a base package?  That would mean that
>> people who wanted just uima base would be downloading a lot more than they
>> will use.
>>     
>
>  I was suggesting that an "everything" package might replace what you
> were suggesting as the "base + uima-as" package.  I haven't completely
> made up my mind about dropping the base-only release.  Would our users
> really be that upset about getting extra stuff they don't want?  (I
> don't think we're talking about huge amounts of stuff to download, are
> we?)
>   
I agree that simpler is better, and bundling things is simpler.  But 
(there's always one of these... ) uima-as is quite a bit "bigger" than 
the base; includes in its distribution the Spring framework, ActiveMQ, 
and Saxon (the XSLT transformer), as well as the custom code for uima-as.
> Furthermore, if we have a lot more tools under development, it may
> make sense to consider tools separate from the "base" - and at that
> point would anyone want a base-only release?
>   
I agree it's simpler / better to have most of the tools in the base - 
which is where we have them now, except for tools which are specific to 
uima-as are packaged with uima-as (this is the extension to the 
component descriptor editor to support editing deployment descriptors).
>   
>> We could, of course, have a style where the binaries would be:
>>  base
>>  base + annotators
>>  base + annotators + simple server
>>  base + uima-as
>>  base + c++
>>
>> etc., various "combinations" of the parts.  I think such a packaging would
>> get out of hand (too many variants).
>> Or, perhaps you're suggesting:
>>  base
>>  base + everything else (or a subset of everything else - e.g., uima-as, cas
>> editor, annotators, but not the simple server?)
>>
>> I think our offerings will continue to grow, and may become more diverse
>> (e.g., a sample "Type System", or corpora for training, or repository
>> functionality might be added).  So I lean toward a strategy that allows for
>> some number of "parts" as separately managed things, together with an
>> attempt to occasionally put out semi-synchronized "big" releases with all or
>> most of the parts, like the Eclipse approach.
>>
>>     
>
> I think having synchronized (why only semi?) "big" releases with all
> the parts (at least all the mature ones) would be a big improvement.
>   
Semi - meant that not all of the components might be in the big 
release.   For instance, if at some point we host test "corpora", those 
might not change very much, and not be in the release.
> As for the piecemeal releases, could these only be for minor version
> updates since the last major release?  
Yes, that was the idea.  Also would be good to allow them as initial 
releases.  So, for instance, if we had 2.2.2 out, and finally the C++ 
version is approved for release, it could be at the 2.2.2 level, but 
released separately.  Following that, we could fold it into the the next 
big release, if we were doing all of it together.
> I can see some value in having
> separate bug-fix updates for different components, sort of like the
> hotfix we just had (but with changing the version #).  The idea is
> that users can avoid doing any update at all if the only thing that's
> changed is something they don't use.  So we're saving the user work,
> rather than adding more work which is what we're doing by not having
> any synchronization at all between our parts and forcing users to
> download things piecemeal as they become available.
>   
I agree.

Perhaps we should separate these 2 ideas:
(1) releasing:  this is a process of packaging, voting, uploading, etc.
(2) packaging: (warning - I'm dreaming here) maybe we could invent some 
new approach whereby the downloader could easily select exactly what 
they wanted from our web site, from a single item to sets of items...

-Marshall

Re: Considering some re-org of SVN

[Adam Lally <al...@alum.rpi.edu>]

On Tue, Jul 8, 2008 at 12:21 PM, Marshall Schor <ms...@schor.com> wrote:
> Also, if I understand you right, you're suggesting that we have just an
> "everything" package, instead of these "add-on" packages. If we have just an
> everything package, would we not have a base package?  That would mean that
> people who wanted just uima base would be downloading a lot more than they
> will use.

 I was suggesting that an "everything" package might replace what you
were suggesting as the "base + uima-as" package.  I haven't completely
made up my mind about dropping the base-only release.  Would our users
really be that upset about getting extra stuff they don't want?  (I
don't think we're talking about huge amounts of stuff to download, are
we?)

Furthermore, if we have a lot more tools under development, it may
make sense to consider tools separate from the "base" - and at that
point would anyone want a base-only release?

> We could, of course, have a style where the binaries would be:
>  base
>  base + annotators
>  base + annotators + simple server
>  base + uima-as
>  base + c++
>
> etc., various "combinations" of the parts.  I think such a packaging would
> get out of hand (too many variants).
> Or, perhaps you're suggesting:
>  base
>  base + everything else (or a subset of everything else - e.g., uima-as, cas
> editor, annotators, but not the simple server?)
>
> I think our offerings will continue to grow, and may become more diverse
> (e.g., a sample "Type System", or corpora for training, or repository
> functionality might be added).  So I lean toward a strategy that allows for
> some number of "parts" as separately managed things, together with an
> attempt to occasionally put out semi-synchronized "big" releases with all or
> most of the parts, like the Eclipse approach.
>

I think having synchronized (why only semi?) "big" releases with all
the parts (at least all the mature ones) would be a big improvement.

As for the piecemeal releases, could these only be for minor version
updates since the last major release?  I can see some value in having
separate bug-fix updates for different components, sort of like the
hotfix we just had (but with changing the version #).  The idea is
that users can avoid doing any update at all if the only thing that's
changed is something they don't use.  So we're saving the user work,
rather than adding more work which is what we're doing by not having
any synchronization at all between our parts and forcing users to
download things piecemeal as they become available.

  -Adam

Re: Considering some re-org of SVN

[Marshall Schor <ms...@schor.com>]

Adam Lally wrote:
> On Wed, Jul 2, 2008 at 3:06 PM, Marshall Schor <ms...@schor.com> wrote:
>   
>> We currently have the following "releases" done or being done or planned:
>>
>> 1 - main uima base (released 2.2.2-incubating)
>> 1a - main uima "hot fix" fp1 (out for a vote)
>> 2 - uima-as (it was not ready when the base was, and is just now out for a
>> vote) <note: this component has a 5D002 export classification>
>> 3 - uima - add-ons (annotators plus the simple server)  (released,
>> 2.2.2-incubating)
>> 4 - cas editor (about to go out for a vote)
>> 5 - C++ (soon to go for a vote)
>>
>>     
>
> Is it possible to have a single release (with a single version number)
> that could have multiple different binary packages (Java framework,
> C++ framework, async scaleout, tools, etc.), where all the parts get
> released and voted on at the same time?  There could be just a single
> source distribution, and then various binaries.  This might make
> things less fragmented and less confusing - for us, the IPMC, and our
> users.
>   
Sure. It has the "pros" that you site.  The "cons" are that it delays 
other parts from being released until the all the parts are ready for 
release.  For instance, the C++ part isn't quite ready for release.  I 
do think that Eclipse is a good model - it has major releases, but the 
other parts do have the ability to do other releases as well.

>   
>> One thing I would like to see at some point going forward is not
>> up-versioning our Jars when they don't change from release to release.  For
>> instance, when we release uima-base, it includes the jvinci jars - which
>> haven't been changed recently (I may be wrong here...), but it would be nice
>> to have that "stability" reflected in the release.  I see that Eclipse often
>> does this in their releases.
>>
>>     
>
> Do you mean not up-versioning the Maven artifacts?  We don't have
> version numbers in the jar file names themselves (at least not yet).
> Maybe this fits with my suggestion above - a single release, but with
> the different jars (or other kinds of parts) marked with appropriate
> version numbers.
>   
Right.
>> Likewise, it would be good to have the uima-as release, which is made to
>> "depend" on a particular base uima release, just include those jars (from
>> Maven, for instance) when creating the distributable "bin" artifact.  (We
>> didn't do that for this last release because at the time we did the build
>> process, uima 2.2.2-incubating wasn't "out" yet).
>>
>>     
>
> Hmmm... would you also do that for the C++ framework too, and the
> tools, etc.?  Instead, how about an "everything" package that includes
> all the stuff?
>   
I think this issue breaks down into the following considerations:

There are source and binary releases.  The source ones have 2 purposes, 
it seems:
   (a) providing the new, additional sources that make up UIMA-AS
   (b) providing something that users can "build" UIMA-AS from
These are not the same, because the current build process depends on 
sources from the base.  We did a compromise for the first release of 
including some of the sources from the base needed for building.   We 
could and maybe should work on the build-from-source process to have it 
be more modular, and try to make (b) work with (a) sources.

Also, if I understand you right, you're suggesting that we have just an 
"everything" package, instead of these "add-on" packages. If we have 
just an everything package, would we not have a base package?  That 
would mean that people who wanted just uima base would be downloading a 
lot more than they will use. 

We could, of course, have a style where the binaries would be:
   base
   base + annotators
   base + annotators + simple server
   base + uima-as
   base + c++

etc., various "combinations" of the parts.  I think such a packaging 
would get out of hand (too many variants). 

Or, perhaps you're suggesting:
  base
  base + everything else (or a subset of everything else - e.g., 
uima-as, cas editor, annotators, but not the simple server?)

I think our offerings will continue to grow, and may become more diverse 
(e.g., a sample "Type System", or corpora for training, or repository 
functionality might be added).  So I lean toward a strategy that allows 
for some number of "parts" as separately managed things, together with 
an attempt to occasionally put out semi-synchronized "big" releases with 
all or most of the parts, like the Eclipse approach.

-Marshall

Re: Considering some re-org of SVN

[Adam Lally <al...@alum.rpi.edu>]

On Wed, Jul 2, 2008 at 3:06 PM, Marshall Schor <ms...@schor.com> wrote:
> We currently have the following "releases" done or being done or planned:
>
> 1 - main uima base (released 2.2.2-incubating)
> 1a - main uima "hot fix" fp1 (out for a vote)
> 2 - uima-as (it was not ready when the base was, and is just now out for a
> vote) <note: this component has a 5D002 export classification>
> 3 - uima - add-ons (annotators plus the simple server)  (released,
> 2.2.2-incubating)
> 4 - cas editor (about to go out for a vote)
> 5 - C++ (soon to go for a vote)
>

Is it possible to have a single release (with a single version number)
that could have multiple different binary packages (Java framework,
C++ framework, async scaleout, tools, etc.), where all the parts get
released and voted on at the same time?  There could be just a single
source distribution, and then various binaries.  This might make
things less fragmented and less confusing - for us, the IPMC, and our
users.

> One thing I would like to see at some point going forward is not
> up-versioning our Jars when they don't change from release to release.  For
> instance, when we release uima-base, it includes the jvinci jars - which
> haven't been changed recently (I may be wrong here...), but it would be nice
> to have that "stability" reflected in the release.  I see that Eclipse often
> does this in their releases.
>

Do you mean not up-versioning the Maven artifacts?  We don't have
version numbers in the jar file names themselves (at least not yet).
Maybe this fits with my suggestion above - a single release, but with
the different jars (or other kinds of parts) marked with appropriate
version numbers.

> Likewise, it would be good to have the uima-as release, which is made to
> "depend" on a particular base uima release, just include those jars (from
> Maven, for instance) when creating the distributable "bin" artifact.  (We
> didn't do that for this last release because at the time we did the build
> process, uima 2.2.2-incubating wasn't "out" yet).
>

Hmmm... would you also do that for the C++ framework too, and the
tools, etc.?  Instead, how about an "everything" package that includes
all the stuff?

  -Adam

Re: Considering some re-org of SVN

[Marshall Schor <ms...@schor.com>]

Thilo Goetz wrote:
> Adam Lally wrote:
>> On Wed, Jul 2, 2008 at 10:34 AM, Michael Baessler
>> <mb...@michael-baessler.de> wrote:
>>> Marshall Schor wrote:
>>>> Possible names for this other node might be: "parts".  Or we might 
>>>> want
>>>> to have several names that categorized the kinds of things - such as
>>>> "annotators", "servers", "embeddings", "corpora", "typeSystems",
>>>> "tools", etc.
>>>>
>>> +1 for having an organization structure based on component type such 
>>> as tools, UIMA core,
>>> annotators, etc. But if we change the structure we should also move 
>>> the current tools we ship with
>>> the UIMA core bundle to have one place where the user gets tools for 
>>> UIMA.
>>>
>>> The separation of UIMA core, tools, annotators also has 
>>> disadvantage. We never come to the point to
>>> have a rich UIMA framework download where some real analytics are 
>>> embedded and work out of the box.
>>> If we want to do that we have to think about a special "analytics 
>>> package" where some pre-configured
>>> analysis is included with a special set of descriptors.
>>>
>>
>> I am already thinking we may have enough different releases happening,
>> if not too many.  There is overhead associated with each release,
>> especially while we're in the incubator and have to bug the IPMC every
>> time we want to release something.  Rather than think about how to
>> restructure SVN, I prefer we first think about how many different
>> artifacts we want to be separately releasing.  Then I'd agree to
>> restructure SVN to suit what we decide.
>
> +1, put the horse before the cart.
I agree with the general sentiments above.

We currently have the following "releases" done or being done or planned:

1 - main uima base (released 2.2.2-incubating)
1a - main uima "hot fix" fp1 (out for a vote)
2 - uima-as (it was not ready when the base was, and is just now out for 
a vote) <note: this component has a 5D002 export classification>
3 - uima - add-ons (annotators plus the simple server)  (released, 
2.2.2-incubating)
4 - cas editor (about to go out for a vote)
5 - C++ (soon to go for a vote)

The best thing, of course, would be to exit the incubator :-) so we 
could plan this with fewer constraints...

One thing I would like to see at some point going forward is not 
up-versioning our Jars when they don't change from release to release.  
For instance, when we release uima-base, it includes the jvinci jars - 
which haven't been changed recently (I may be wrong here...), but it 
would be nice to have that "stability" reflected in the release.  I see 
that Eclipse often does this in their releases.

Likewise, it would be good to have the uima-as release, which is made to 
"depend" on a particular base uima release, just include those jars 
(from Maven, for instance) when creating the distributable "bin" 
artifact.  (We didn't do that for this last release because at the time 
we did the build process, uima 2.2.2-incubating wasn't "out" yet).

-Marshall
>
>>
>> If we take the current tools out of the current SDK download, we make
>> it a lot less useful.  I think it's a better out of the box experience
>> if they get one download that includes good tooling.  So I'd prefer
>> seeing mature sandbox tools get integrated into the SDK release rather
>> than be a separate package.
>>
>> For annotators, I'm also leaning towards having some good general
>> purpose annotators included in the SDK.  If they start to get more
>> specialized, then I am not so sure.
>>
>> Of course this may lead to a bigger download, but my personal feeling
>> is that this isn't much of a concern.  If we start seeing problems or
>> getting complaints that our release is too big, maybe we go back and
>> consider splitting it at that point.
>>
>>   -Adam
>
>

Re: Considering some re-org of SVN

[Thilo Goetz <tw...@gmx.de>]

Adam Lally wrote:
> On Wed, Jul 2, 2008 at 10:34 AM, Michael Baessler
> <mb...@michael-baessler.de> wrote:
>> Marshall Schor wrote:
>>> Possible names for this other node might be: "parts".  Or we might want
>>> to have several names that categorized the kinds of things - such as
>>> "annotators", "servers", "embeddings", "corpora", "typeSystems",
>>> "tools", etc.
>>>
>> +1 for having an organization structure based on component type such as tools, UIMA core,
>> annotators, etc. But if we change the structure we should also move the current tools we ship with
>> the UIMA core bundle to have one place where the user gets tools for UIMA.
>>
>> The separation of UIMA core, tools, annotators also has disadvantage. We never come to the point to
>> have a rich UIMA framework download where some real analytics are embedded and work out of the box.
>> If we want to do that we have to think about a special "analytics package" where some pre-configured
>> analysis is included with a special set of descriptors.
>>
> 
> I am already thinking we may have enough different releases happening,
> if not too many.  There is overhead associated with each release,
> especially while we're in the incubator and have to bug the IPMC every
> time we want to release something.  Rather than think about how to
> restructure SVN, I prefer we first think about how many different
> artifacts we want to be separately releasing.  Then I'd agree to
> restructure SVN to suit what we decide.

+1, put the horse before the cart.

> 
> If we take the current tools out of the current SDK download, we make
> it a lot less useful.  I think it's a better out of the box experience
> if they get one download that includes good tooling.  So I'd prefer
> seeing mature sandbox tools get integrated into the SDK release rather
> than be a separate package.
> 
> For annotators, I'm also leaning towards having some good general
> purpose annotators included in the SDK.  If they start to get more
> specialized, then I am not so sure.
> 
> Of course this may lead to a bigger download, but my personal feeling
> is that this isn't much of a concern.  If we start seeing problems or
> getting complaints that our release is too big, maybe we go back and
> consider splitting it at that point.
> 
>   -Adam

Re: Considering some re-org of SVN

[Adam Lally <al...@alum.rpi.edu>]

On Wed, Jul 2, 2008 at 10:34 AM, Michael Baessler
<mb...@michael-baessler.de> wrote:
> Marshall Schor wrote:
>> Possible names for this other node might be: "parts".  Or we might want
>> to have several names that categorized the kinds of things - such as
>> "annotators", "servers", "embeddings", "corpora", "typeSystems",
>> "tools", etc.
>>
> +1 for having an organization structure based on component type such as tools, UIMA core,
> annotators, etc. But if we change the structure we should also move the current tools we ship with
> the UIMA core bundle to have one place where the user gets tools for UIMA.
>
> The separation of UIMA core, tools, annotators also has disadvantage. We never come to the point to
> have a rich UIMA framework download where some real analytics are embedded and work out of the box.
> If we want to do that we have to think about a special "analytics package" where some pre-configured
> analysis is included with a special set of descriptors.
>

I am already thinking we may have enough different releases happening,
if not too many.  There is overhead associated with each release,
especially while we're in the incubator and have to bug the IPMC every
time we want to release something.  Rather than think about how to
restructure SVN, I prefer we first think about how many different
artifacts we want to be separately releasing.  Then I'd agree to
restructure SVN to suit what we decide.

If we take the current tools out of the current SDK download, we make
it a lot less useful.  I think it's a better out of the box experience
if they get one download that includes good tooling.  So I'd prefer
seeing mature sandbox tools get integrated into the SDK release rather
than be a separate package.

For annotators, I'm also leaning towards having some good general
purpose annotators included in the SDK.  If they start to get more
specialized, then I am not so sure.

Of course this may lead to a bigger download, but my personal feeling
is that this isn't much of a concern.  If we start seeing problems or
getting complaints that our release is too big, maybe we go back and
consider splitting it at that point.

  -Adam

Re: Considering some re-org of SVN

[Michael Baessler <mb...@michael-baessler.de>]

Marshall Schor wrote:
> I'm thinking about some re-org of our SVN layout based on these
> observations
> 
> -We're getting a lot of components.  Many of these are on somewhat
> different release cycles.
> 
> -We initially had a "main" node, a cpp node, a site note and a sandbox
> node.  The sandbox was for new-ish things.  As some of these things get
> more "mainline" - it would make sense to have them perhaps under another
> node to indicate that.  The idea would be that things in the sandbox
> were user-beware, but things in this other node were more "dependable"
> and "proven".
> Possible names for this other node might be: "parts".  Or we might want
> to have several names that categorized the kinds of things - such as
> "annotators", "servers", "embeddings", "corpora", "typeSystems",
> "tools", etc.
> 
> -The SVN conventions lean toward having branches and tags which are one
> level above the thing being released.  Right now, for sandbox projects,
> these are 2 levels above the released thing.  I think that, going
> forward, it would be better to go with the convention, following the
> convention-over-configuration philosophy, because the components are not
> likely to all be released on the same release cycle (although that would
> be a nice "goal" - like Eclipse does with it's many-part major releases).
> 
> Other opinions?
> 
> -Marshall
> 
> 
+1 for having an organization structure based on component type such as tools, UIMA core,
annotators, etc. But if we change the structure we should also move the current tools we ship with
the UIMA core bundle to have one place where the user gets tools for UIMA.

The separation of UIMA core, tools, annotators also has disadvantage. We never come to the point to
have a rich UIMA framework download where some real analytics are embedded and work out of the box.
If we want to do that we have to think about a special "analytics package" where some pre-configured
analysis is included with a special set of descriptors.

-- Michael