Not quite a war

[Karl Wright <kw...@metacarta.com>]

I have been working on a preliminary set of build.xml files for the various components of lcf, as per Grant's suggestion, and 
have been following (roughly) the packaging setup of the granted code.  There is, however, a detail we should discuss.

The framework itself consists of an "agents" process, and two Tomcat web applications.  One of these web applications is the 
authority service (authority.war).  The other web application (crawler.war) is for the crawler UI, which is what I'd like to 
focus on for the purposes of this discussion.

Each connector provides one or more jars for the agents process, all of which need to be available to the crawler UI web 
application.  Each connector ALSO provides specific jsp elements that are meant to be part of the crawler UI.  In the MetaCarta 
world, we dealt with this by building a special runt "war" for each connector, which isn't really a full war but is meant to 
represent the delta needed to augment crawler.war.  So when you install an individual connector, it effectively unpacks 
crawler.war, unpacks randomconnector.war on top of it, and then packs crawler.war back up.

The question is: Is this an acceptable way of packaging an individual connector in Apache Land?  Or do we want to do something 
different?  I suppose we might want to call these connector war's by some other extension?  Thoughts?  Ideas?

Karl

Re: Not quite a war

[Karl Wright <kw...@metacarta.com>]

Grant Ingersoll wrote:
> On Feb 23, 2010, at 10:50 AM, Karl Wright wrote:
> 
>> Grant Ingersoll wrote:
>>> On Feb 12, 2010, at 4:44 AM, Karl Wright wrote:
>>>> I have been working on a preliminary set of build.xml files for the various components of lcf, as per Grant's suggestion, and have been following (roughly) the packaging setup of the granted code.  There is, however, a detail we should discuss.
>>>>
>>>> The framework itself consists of an "agents" process, and two Tomcat web applications.  One of these web applications is the authority service (authority.war).  The other web application (crawler.war) is for the crawler UI, which is what I'd like to focus on for the purposes of this discussion.
>>>>
>>>> Each connector provides one or more jars for the agents process, all of which need to be available to the crawler UI web application.  Each connector ALSO provides specific jsp elements that are meant to be part of the crawler UI.  In the MetaCarta world, we dealt with this by building a special runt "war" for each connector, which isn't really a full war but is meant to represent the delta needed to augment crawler.war.  So when you install an individual connector, it effectively unpacks crawler.war, unpacks randomconnector.war on top of it, and then packs crawler.war back up.
>>>>
>>> Seems fine to me, but could we also have ANT create all the various versions plus an uber one? 
>> That's exactly what the ant script under modules does.
>>
>>>> The question is: Is this an acceptable way of packaging an individual connector in Apache Land?  Or do we want to do something different?  I suppose we might want to call these connector war's by some other extension?  Thoughts?  Ideas?
>>> How big is the file if we just put all of them together?  Presumably, people have multiple connectors, right, so that would mean adding several overlays, right?  I guess I'd lean for simple to start and just create one big one.
>> It's about 20MB, all told.
> 
> I'd say we don't even worry about it then.  That's tiny.  People can add custom packaging later if they want.
> 

The MetaCarta packaging requires them to be kept separate, so I have a separate ant build for each connector.  I strongly hope 
we keep that structure.  We did it that way for mainly licensing reasons, as you might imagine, although maintenance releases 
are also easier if they are separated in this way.

There are also separate jars that are built that way because there's an explicit need.  For example, the documentum client API 
and filenet API both rely on specific versions of open-source packages, and in documentum's case also rely on native code.  So 
there's a separate server process that can wall that funkiness off.  That means separate jars as well.

The ant build in place is currently structured so that:

- there's a separate ant build for the framework, and for each individual connector
- there's an ant build at the "modules" level that is meant to run all of the individual ones, and combine things appropriately.

The module-level build currently gloms together all the wars and iars into a pair of war files.  But it leaves the process-level 
jars separate, and throws them into directories consistent with their process affinity.  More combination could be done, 
although I think this is pretty straightforward even now.  See what you think.  You can read 
http://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+Lucene+Connector+Framework for all the details.

Thanks,
Karl

Re: Not quite a war

[Grant Ingersoll <gs...@apache.org>]

On Feb 23, 2010, at 10:50 AM, Karl Wright wrote:

> Grant Ingersoll wrote:
>> On Feb 12, 2010, at 4:44 AM, Karl Wright wrote:
>>> I have been working on a preliminary set of build.xml files for the various components of lcf, as per Grant's suggestion, and have been following (roughly) the packaging setup of the granted code.  There is, however, a detail we should discuss.
>>> 
>>> The framework itself consists of an "agents" process, and two Tomcat web applications.  One of these web applications is the authority service (authority.war).  The other web application (crawler.war) is for the crawler UI, which is what I'd like to focus on for the purposes of this discussion.
>>> 
>>> Each connector provides one or more jars for the agents process, all of which need to be available to the crawler UI web application.  Each connector ALSO provides specific jsp elements that are meant to be part of the crawler UI.  In the MetaCarta world, we dealt with this by building a special runt "war" for each connector, which isn't really a full war but is meant to represent the delta needed to augment crawler.war.  So when you install an individual connector, it effectively unpacks crawler.war, unpacks randomconnector.war on top of it, and then packs crawler.war back up.
>>> 
>> Seems fine to me, but could we also have ANT create all the various versions plus an uber one? 
> 
> That's exactly what the ant script under modules does.
> 
>>> The question is: Is this an acceptable way of packaging an individual connector in Apache Land?  Or do we want to do something different?  I suppose we might want to call these connector war's by some other extension?  Thoughts?  Ideas?
>> How big is the file if we just put all of them together?  Presumably, people have multiple connectors, right, so that would mean adding several overlays, right?  I guess I'd lean for simple to start and just create one big one.
> 
> It's about 20MB, all told.

I'd say we don't even worry about it then.  That's tiny.  People can add custom packaging later if they want.

Re: Not quite a war

[Karl Wright <kw...@metacarta.com>]

Grant Ingersoll wrote:
> On Feb 12, 2010, at 4:44 AM, Karl Wright wrote:
> 
>> I have been working on a preliminary set of build.xml files for the various components of lcf, as per Grant's suggestion, and have been following (roughly) the packaging setup of the granted code.  There is, however, a detail we should discuss.
>>
>> The framework itself consists of an "agents" process, and two Tomcat web applications.  One of these web applications is the authority service (authority.war).  The other web application (crawler.war) is for the crawler UI, which is what I'd like to focus on for the purposes of this discussion.
>>
>> Each connector provides one or more jars for the agents process, all of which need to be available to the crawler UI web application.  Each connector ALSO provides specific jsp elements that are meant to be part of the crawler UI.  In the MetaCarta world, we dealt with this by building a special runt "war" for each connector, which isn't really a full war but is meant to represent the delta needed to augment crawler.war.  So when you install an individual connector, it effectively unpacks crawler.war, unpacks randomconnector.war on top of it, and then packs crawler.war back up.
>>
> 
> Seems fine to me, but could we also have ANT create all the various versions plus an uber one? 
> 

That's exactly what the ant script under modules does.

> 
>> The question is: Is this an acceptable way of packaging an individual connector in Apache Land?  Or do we want to do something different?  I suppose we might want to call these connector war's by some other extension?  Thoughts?  Ideas?
> 
> 
> How big is the file if we just put all of them together?  Presumably, people have multiple connectors, right, so that would mean adding several overlays, right?  I guess I'd lean for simple to start and just create one big one.
> 
> 
> 

It's about 20MB, all told.
Karl

Re: Not quite a war

[Grant Ingersoll <gs...@apache.org>]

On Feb 12, 2010, at 4:44 AM, Karl Wright wrote:

> I have been working on a preliminary set of build.xml files for the various components of lcf, as per Grant's suggestion, and have been following (roughly) the packaging setup of the granted code.  There is, however, a detail we should discuss.
> 
> The framework itself consists of an "agents" process, and two Tomcat web applications.  One of these web applications is the authority service (authority.war).  The other web application (crawler.war) is for the crawler UI, which is what I'd like to focus on for the purposes of this discussion.
> 
> Each connector provides one or more jars for the agents process, all of which need to be available to the crawler UI web application.  Each connector ALSO provides specific jsp elements that are meant to be part of the crawler UI.  In the MetaCarta world, we dealt with this by building a special runt "war" for each connector, which isn't really a full war but is meant to represent the delta needed to augment crawler.war.  So when you install an individual connector, it effectively unpacks crawler.war, unpacks randomconnector.war on top of it, and then packs crawler.war back up.
> 

Seems fine to me, but could we also have ANT create all the various versions plus an uber one? 


> The question is: Is this an acceptable way of packaging an individual connector in Apache Land?  Or do we want to do something different?  I suppose we might want to call these connector war's by some other extension?  Thoughts?  Ideas?


How big is the file if we just put all of them together?  Presumably, people have multiple connectors, right, so that would mean adding several overlays, right?  I guess I'd lean for simple to start and just create one big one.