You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Shinichiro Abe <sh...@gmail.com> on 2014/09/25 19:31:07 UTC
MCF 2.x binary package directory structure
Hi dev,
Currently top folders in dist are too many, I think.
Do you want to gather folders per roles?
I'd like to change MCF dist directory.
for instance:
connector-specific-processes
L documentum-registry-process
L documentum-server-process
L filenet-registry-process
L filenet-registry-process
connectors
L connector-lib
L connector-lib-proprietary
L connectors.xml
L connectors-proprietary.xml
docs
lib
L lib
L lib-proprietary
plugins
L elasticsearch-integration
L meridio-integration
L sharepoint-integration
L solr-integration
process-multi
L multiprocess-file-example
L multiprocess-file-example-proprietary
L multiprocess-zk-example
L multiprocess-zk-example-proprietary
process-single
L example
L example-proprietary
script-engine
webapps
L web
L web-proprietary
README.txt etc
My points are:
1. connector-specific-processes and plugins should not be top dir.
Specific users only will use them.
2. It is better to separate Jetty base and multi process dirs, new MCF
users will be aware of the role of process when unzipping binary
package.
3. I think we may remove start.jar and replace it with start.sh
because build.xml in framework is currently too long to include class
paths in lib. Also, jar files name in lib should have version, we
could not know one unless opening build.xml from source package.
4. If we don't remove start.jar, in process-single directory we can
create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
dir(for logging.ini) and etc dir(for jetty.xml). Then we could upgrade
Jetty, customize configuration, and use arguments which start.jar has
e.g STOP.KEY, STOP.PORT --stop etc.
Regards,
Shinichiro Abe
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
Hi Abe-san,
For any of the three tickets related to your original post
(CONNECTORS-1048, CONNECTORS-345, CONNECTORS-1049), if you want to assign
the ticket to yourself, and propose an official patch for just that issue,
I'll be happy to review it and test it out. Otherwise, I'll try to get to
these sometime soon.
Karl
On Fri, Sep 26, 2014 at 6:33 AM, Karl Wright <da...@gmail.com> wrote:
> Created CONNECTORS-1049 to describe relocating plugins and connector
> processes.
> Karl
>
> On Fri, Sep 26, 2014 at 5:28 AM, Karl Wright <da...@gmail.com> wrote:
>
>> I've replied to the jetty vs. tomcat issue in the CONNECTORS-345 comment
>> thread.
>> Thanks!
>> Karl
>>
>> On Fri, Sep 26, 2014 at 5:22 AM, Karl Wright <da...@gmail.com> wrote:
>>
>>> I've moved CONNECTORS-345 to "Fix in 2.0" and attached your comments.
>>> Let's talk about the jetty deployment issues in that ticket thread.
>>>
>>> Karl
>>>
>>>
>>> On Fri, Sep 26, 2014 at 5:16 AM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> CONNECTORS-1048 created for including jar versions.
>>>> Karl
>>>>
>>>> On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <da...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Abe-san,
>>>>>
>>>>> It seems to me that we are discussing some distinct and independent
>>>>> improvements, so maybe I will open specific tickets for each one and we can
>>>>> work on things one at a time. But, first:
>>>>>
>>>>> bq. I just simplified directories since other Apache projects look
>>>>> like much simpler.
>>>>>
>>>>> That's true, but I don't know of any other Apache project which
>>>>> attempts to handle conditional compilation to the degree that we have to in
>>>>> ManifoldCF. So I think we should expect things to be more complicated. :-)
>>>>>
>>>>> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>>> other), to create *-proprietary directories is awkward, isn't it?
>>>>>
>>>>> I've never particularly liked the -proprietary paradigm, but I could
>>>>> think of no other way. The reasons behind it was that we *must* legally
>>>>> exclude anything proprietary from the distribution. By making sure that
>>>>> every proprietary jar is in a well-labeled directory, I can simply make
>>>>> sure all such directories are excluded. But that also includes wars.
>>>>> Since wars must include all jars, that meant that the build had to create
>>>>> two different sets of wars, and put them in different directories. The
>>>>> only other alternative is to guarantee that we always build our release
>>>>> candidates from a fresh checkout each time, and that seemed risky.
>>>>>
>>>>> I agree that it would be more convenient for the user if our build
>>>>> created one set of wars based on what was included by the user, and that we
>>>>> shipped just the minimal binary that didn't include proprietary code. If
>>>>> we can solve the war problem, we can do that.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
>>>>> shinichiro.abe.1@gmail.com> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> Thanks for the reply.
>>>>>>
>>>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>>>> framework is
>>>>>> > currently too long to include class
>>>>>> > paths in lib". Can you clarify?
>>>>>>
>>>>>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>>>>>
>>>>>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>>>>>> If new jars introduced with new feature, manifest-cp-proprietary must
>>>>>> be tweaked, too.
>>>>>> (i18n properties files also bother us.)
>>>>>>
>>>>>> Alternatively, the ant task below is useful.
>>>>>>
>>>>>> common-build.xml: Add artifact-version to dest.
>>>>>> + <get
>>>>>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>>>>>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>>>>>
>>>>>> framework/build.xml: New manifest-cp property setting works fine.
>>>>>>
>>>>>> - <property name="manifest-cp-74" value="${manifest-cp-73}
>>>>>> ../lib/xmlsec.jar"/>
>>>>>> - <property name="manifest-cp-75" value="${manifest-cp-74}
>>>>>> ../lib/opensaml.jar"/>
>>>>>>
>>>>>> + <property name="liblocation" location="../lib" />
>>>>>> + <path id="lib-jars">
>>>>>> + <fileset dir="${liblocation}" includes="*.jar"/>
>>>>>> + </path>
>>>>>> + <pathconvert property="manifest-cp" refid="lib-jars"
>>>>>> targetos="unix" pathsep=" ">
>>>>>> + <map from="${liblocation}" to="../lib"/>
>>>>>> + <map from="\" to="/"/>
>>>>>> + </pathconvert>
>>>>>>
>>>>>> + <!-- <property name="manifest-cp"
>>>>>> value="${manifest-cp-75}"/> -->
>>>>>>
>>>>>> > (2) Some of these suggestions seem to be making distinctions
>>>>>> between files
>>>>>> > and directories that I don't understand the reason for.
>>>>>>
>>>>>> 1. I came to mind CONNECTORS-345.
>>>>>> 2. Currently I have to create scripts in production:
>>>>>> startup.sh --
>>>>>> java -jar start.jar
>>>>>> echo $! > mcf.pid
>>>>>> shutdown.sh --
>>>>>> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>>>>>> agent-shutdown AFAIK but this script isn't cool.
>>>>>>
>>>>>> 3. start.jar command line options are useful.
>>>>>>
>>>>>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>>>>>
>>>>>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat.
>>>>>> Of course combined.war can do that though.
>>>>>> I thought additional jars could copy to another directory, and might
>>>>>> load these using Custom ClassLoader.
>>>>>>
>>>>>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>>>>>
>>>>>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>>>>>> requestHeaderSize and so on in jetty.xml --
>>>>>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>>>>>> I will drop these suggestions.
>>>>>>
>>>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>>>> other
>>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>>> under
>>>>>> > additional levels of hierarchy. I think it should be possible
>>>>>> immediately
>>>>>> > for a user to see what examples are available. But maybe we could
>>>>>> change
>>>>>> > the names to be clearer.
>>>>>> Yes. Especially I want to change name *-proprietary into something
>>>>>> like *-extra because "proprietary".length() is long.
>>>>>>
>>>>>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>>>> other), to create *-proprietary directories is awkward, isn't it?
>>>>>> I know CONNECTORS-402 intent, but more simplified, can we say that
>>>>>> "if you use these jars,
>>>>>> please get source and run ''ant make-deps build"? -- I usually say
>>>>>> so, no problems occur for now.
>>>>>>
>>>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>>>> > yyy-proprietary, it would be more appropriate to have a
>>>>>> "proprietary"
>>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>>>
>>>>>> Ok.
>>>>>> But I want to know, can we deprecate example and move
>>>>>> example-proprietary to example?
>>>>>> I'd like to merge these examples. If proprietary jars exists, MCF
>>>>>> just have to load from extra-dir.
>>>>>> I assume that we can have extra-lib which is optional, If it exists,
>>>>>> MCF will load any Jars
>>>>>> found in this directory and use them to resolve connectors specified
>>>>>> in connectors.xml.
>>>>>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>>>>>
>>>>>> I just simplified directories since other Apache projects look like
>>>>>> much simpler.
>>>>>>
>>>>>> Regards,
>>>>>> Shinichiro Abe.
>>>>>>
>>>>>> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>>>>>>
>>>>>> > Hi Abe-san,
>>>>>> >
>>>>>> > Some comments:
>>>>>> >
>>>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>>>> framework is
>>>>>> > currently too long to include class
>>>>>> > paths in lib". Can you clarify?
>>>>>> >
>>>>>> > (2) Some of these suggestions seem to be making distinctions
>>>>>> between files
>>>>>> > and directories that I don't understand the reason for. For
>>>>>> example: "in
>>>>>> > process-single directory we can
>>>>>> > create lib dir(for jetty jars), lib/ext dir(for logger jars),
>>>>>> resource
>>>>>> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate
>>>>>> jars in
>>>>>> > this way? They are all necessary at the root level to run the
>>>>>> example. I
>>>>>> > would not understand where to add a new jar with this arrangement
>>>>>> because
>>>>>> > the meaning of the directories is unclear.
>>>>>> >
>>>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>>>> other
>>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>>> under
>>>>>> > additional levels of hierarchy. I think it should be possible
>>>>>> immediately
>>>>>> > for a user to see what examples are available. But maybe we could
>>>>>> change
>>>>>> > the names to be clearer.
>>>>>> >
>>>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>>>> > yyy-proprietary, it would be more appropriate to have a
>>>>>> "proprietary"
>>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>>> >
>>>>>> > (5) Putting version numbers on jars is difficult in some cases,
>>>>>> especially
>>>>>> > in construction of start.jar, because the ant methods for
>>>>>> constructing
>>>>>> > start.jar are poor. The version of each jar would need to be
>>>>>> defined
>>>>>> > globally in the ant build, and included whenever the jar is
>>>>>> referenced.
>>>>>> >
>>>>>> > Karl
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
Created CONNECTORS-1049 to describe relocating plugins and connector
processes.
Karl
On Fri, Sep 26, 2014 at 5:28 AM, Karl Wright <da...@gmail.com> wrote:
> I've replied to the jetty vs. tomcat issue in the CONNECTORS-345 comment
> thread.
> Thanks!
> Karl
>
> On Fri, Sep 26, 2014 at 5:22 AM, Karl Wright <da...@gmail.com> wrote:
>
>> I've moved CONNECTORS-345 to "Fix in 2.0" and attached your comments.
>> Let's talk about the jetty deployment issues in that ticket thread.
>>
>> Karl
>>
>>
>> On Fri, Sep 26, 2014 at 5:16 AM, Karl Wright <da...@gmail.com> wrote:
>>
>>> CONNECTORS-1048 created for including jar versions.
>>> Karl
>>>
>>> On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <da...@gmail.com> wrote:
>>>
>>>> Hi Abe-san,
>>>>
>>>> It seems to me that we are discussing some distinct and independent
>>>> improvements, so maybe I will open specific tickets for each one and we can
>>>> work on things one at a time. But, first:
>>>>
>>>> bq. I just simplified directories since other Apache projects look like
>>>> much simpler.
>>>>
>>>> That's true, but I don't know of any other Apache project which
>>>> attempts to handle conditional compilation to the degree that we have to in
>>>> ManifoldCF. So I think we should expect things to be more complicated. :-)
>>>>
>>>> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>> other), to create *-proprietary directories is awkward, isn't it?
>>>>
>>>> I've never particularly liked the -proprietary paradigm, but I could
>>>> think of no other way. The reasons behind it was that we *must* legally
>>>> exclude anything proprietary from the distribution. By making sure that
>>>> every proprietary jar is in a well-labeled directory, I can simply make
>>>> sure all such directories are excluded. But that also includes wars.
>>>> Since wars must include all jars, that meant that the build had to create
>>>> two different sets of wars, and put them in different directories. The
>>>> only other alternative is to guarantee that we always build our release
>>>> candidates from a fresh checkout each time, and that seemed risky.
>>>>
>>>> I agree that it would be more convenient for the user if our build
>>>> created one set of wars based on what was included by the user, and that we
>>>> shipped just the minimal binary that didn't include proprietary code. If
>>>> we can solve the war problem, we can do that.
>>>>
>>>> Karl
>>>>
>>>>
>>>>
>>>> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
>>>> shinichiro.abe.1@gmail.com> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>> Thanks for the reply.
>>>>>
>>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>>> framework is
>>>>> > currently too long to include class
>>>>> > paths in lib". Can you clarify?
>>>>>
>>>>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>>>>
>>>>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>>>>> If new jars introduced with new feature, manifest-cp-proprietary must
>>>>> be tweaked, too.
>>>>> (i18n properties files also bother us.)
>>>>>
>>>>> Alternatively, the ant task below is useful.
>>>>>
>>>>> common-build.xml: Add artifact-version to dest.
>>>>> + <get
>>>>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>>>>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>>>>
>>>>> framework/build.xml: New manifest-cp property setting works fine.
>>>>>
>>>>> - <property name="manifest-cp-74" value="${manifest-cp-73}
>>>>> ../lib/xmlsec.jar"/>
>>>>> - <property name="manifest-cp-75" value="${manifest-cp-74}
>>>>> ../lib/opensaml.jar"/>
>>>>>
>>>>> + <property name="liblocation" location="../lib" />
>>>>> + <path id="lib-jars">
>>>>> + <fileset dir="${liblocation}" includes="*.jar"/>
>>>>> + </path>
>>>>> + <pathconvert property="manifest-cp" refid="lib-jars"
>>>>> targetos="unix" pathsep=" ">
>>>>> + <map from="${liblocation}" to="../lib"/>
>>>>> + <map from="\" to="/"/>
>>>>> + </pathconvert>
>>>>>
>>>>> + <!-- <property name="manifest-cp" value="${manifest-cp-75}"/>
>>>>> -->
>>>>>
>>>>> > (2) Some of these suggestions seem to be making distinctions between
>>>>> files
>>>>> > and directories that I don't understand the reason for.
>>>>>
>>>>> 1. I came to mind CONNECTORS-345.
>>>>> 2. Currently I have to create scripts in production:
>>>>> startup.sh --
>>>>> java -jar start.jar
>>>>> echo $! > mcf.pid
>>>>> shutdown.sh --
>>>>> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>>>>> agent-shutdown AFAIK but this script isn't cool.
>>>>>
>>>>> 3. start.jar command line options are useful.
>>>>>
>>>>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>>>>
>>>>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat.
>>>>> Of course combined.war can do that though.
>>>>> I thought additional jars could copy to another directory, and might
>>>>> load these using Custom ClassLoader.
>>>>>
>>>>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>>>>
>>>>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>>>>> requestHeaderSize and so on in jetty.xml --
>>>>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>>>>> I will drop these suggestions.
>>>>>
>>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>>> other
>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>> under
>>>>> > additional levels of hierarchy. I think it should be possible
>>>>> immediately
>>>>> > for a user to see what examples are available. But maybe we could
>>>>> change
>>>>> > the names to be clearer.
>>>>> Yes. Especially I want to change name *-proprietary into something
>>>>> like *-extra because "proprietary".length() is long.
>>>>>
>>>>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>>> other), to create *-proprietary directories is awkward, isn't it?
>>>>> I know CONNECTORS-402 intent, but more simplified, can we say that "if
>>>>> you use these jars,
>>>>> please get source and run ''ant make-deps build"? -- I usually say so,
>>>>> no problems occur for now.
>>>>>
>>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>>
>>>>> Ok.
>>>>> But I want to know, can we deprecate example and move
>>>>> example-proprietary to example?
>>>>> I'd like to merge these examples. If proprietary jars exists, MCF just
>>>>> have to load from extra-dir.
>>>>> I assume that we can have extra-lib which is optional, If it exists,
>>>>> MCF will load any Jars
>>>>> found in this directory and use them to resolve connectors specified
>>>>> in connectors.xml.
>>>>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>>>>
>>>>> I just simplified directories since other Apache projects look like
>>>>> much simpler.
>>>>>
>>>>> Regards,
>>>>> Shinichiro Abe.
>>>>>
>>>>> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>>>>>
>>>>> > Hi Abe-san,
>>>>> >
>>>>> > Some comments:
>>>>> >
>>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>>> framework is
>>>>> > currently too long to include class
>>>>> > paths in lib". Can you clarify?
>>>>> >
>>>>> > (2) Some of these suggestions seem to be making distinctions between
>>>>> files
>>>>> > and directories that I don't understand the reason for. For
>>>>> example: "in
>>>>> > process-single directory we can
>>>>> > create lib dir(for jetty jars), lib/ext dir(for logger jars),
>>>>> resource
>>>>> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars
>>>>> in
>>>>> > this way? They are all necessary at the root level to run the
>>>>> example. I
>>>>> > would not understand where to add a new jar with this arrangement
>>>>> because
>>>>> > the meaning of the directories is unclear.
>>>>> >
>>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>>> other
>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>> under
>>>>> > additional levels of hierarchy. I think it should be possible
>>>>> immediately
>>>>> > for a user to see what examples are available. But maybe we could
>>>>> change
>>>>> > the names to be clearer.
>>>>> >
>>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>> >
>>>>> > (5) Putting version numbers on jars is difficult in some cases,
>>>>> especially
>>>>> > in construction of start.jar, because the ant methods for
>>>>> constructing
>>>>> > start.jar are poor. The version of each jar would need to be defined
>>>>> > globally in the ant build, and included whenever the jar is
>>>>> referenced.
>>>>> >
>>>>> > Karl
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
I've replied to the jetty vs. tomcat issue in the CONNECTORS-345 comment
thread.
Thanks!
Karl
On Fri, Sep 26, 2014 at 5:22 AM, Karl Wright <da...@gmail.com> wrote:
> I've moved CONNECTORS-345 to "Fix in 2.0" and attached your comments.
> Let's talk about the jetty deployment issues in that ticket thread.
>
> Karl
>
>
> On Fri, Sep 26, 2014 at 5:16 AM, Karl Wright <da...@gmail.com> wrote:
>
>> CONNECTORS-1048 created for including jar versions.
>> Karl
>>
>> On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <da...@gmail.com> wrote:
>>
>>> Hi Abe-san,
>>>
>>> It seems to me that we are discussing some distinct and independent
>>> improvements, so maybe I will open specific tickets for each one and we can
>>> work on things one at a time. But, first:
>>>
>>> bq. I just simplified directories since other Apache projects look like
>>> much simpler.
>>>
>>> That's true, but I don't know of any other Apache project which attempts
>>> to handle conditional compilation to the degree that we have to in
>>> ManifoldCF. So I think we should expect things to be more complicated. :-)
>>>
>>> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>> other), to create *-proprietary directories is awkward, isn't it?
>>>
>>> I've never particularly liked the -proprietary paradigm, but I could
>>> think of no other way. The reasons behind it was that we *must* legally
>>> exclude anything proprietary from the distribution. By making sure that
>>> every proprietary jar is in a well-labeled directory, I can simply make
>>> sure all such directories are excluded. But that also includes wars.
>>> Since wars must include all jars, that meant that the build had to create
>>> two different sets of wars, and put them in different directories. The
>>> only other alternative is to guarantee that we always build our release
>>> candidates from a fresh checkout each time, and that seemed risky.
>>>
>>> I agree that it would be more convenient for the user if our build
>>> created one set of wars based on what was included by the user, and that we
>>> shipped just the minimal binary that didn't include proprietary code. If
>>> we can solve the war problem, we can do that.
>>>
>>> Karl
>>>
>>>
>>>
>>> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
>>> shinichiro.abe.1@gmail.com> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>> Thanks for the reply.
>>>>
>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>> framework is
>>>> > currently too long to include class
>>>> > paths in lib". Can you clarify?
>>>>
>>>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>>>
>>>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>>>> If new jars introduced with new feature, manifest-cp-proprietary must
>>>> be tweaked, too.
>>>> (i18n properties files also bother us.)
>>>>
>>>> Alternatively, the ant task below is useful.
>>>>
>>>> common-build.xml: Add artifact-version to dest.
>>>> + <get
>>>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>>>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>>>
>>>> framework/build.xml: New manifest-cp property setting works fine.
>>>>
>>>> - <property name="manifest-cp-74" value="${manifest-cp-73}
>>>> ../lib/xmlsec.jar"/>
>>>> - <property name="manifest-cp-75" value="${manifest-cp-74}
>>>> ../lib/opensaml.jar"/>
>>>>
>>>> + <property name="liblocation" location="../lib" />
>>>> + <path id="lib-jars">
>>>> + <fileset dir="${liblocation}" includes="*.jar"/>
>>>> + </path>
>>>> + <pathconvert property="manifest-cp" refid="lib-jars"
>>>> targetos="unix" pathsep=" ">
>>>> + <map from="${liblocation}" to="../lib"/>
>>>> + <map from="\" to="/"/>
>>>> + </pathconvert>
>>>>
>>>> + <!-- <property name="manifest-cp" value="${manifest-cp-75}"/>
>>>> -->
>>>>
>>>> > (2) Some of these suggestions seem to be making distinctions between
>>>> files
>>>> > and directories that I don't understand the reason for.
>>>>
>>>> 1. I came to mind CONNECTORS-345.
>>>> 2. Currently I have to create scripts in production:
>>>> startup.sh --
>>>> java -jar start.jar
>>>> echo $! > mcf.pid
>>>> shutdown.sh --
>>>> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>>>> agent-shutdown AFAIK but this script isn't cool.
>>>>
>>>> 3. start.jar command line options are useful.
>>>>
>>>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>>>
>>>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat. Of
>>>> course combined.war can do that though.
>>>> I thought additional jars could copy to another directory, and might
>>>> load these using Custom ClassLoader.
>>>>
>>>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>>>
>>>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>>>> requestHeaderSize and so on in jetty.xml --
>>>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>>>> I will drop these suggestions.
>>>>
>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>> other
>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>> under
>>>> > additional levels of hierarchy. I think it should be possible
>>>> immediately
>>>> > for a user to see what examples are available. But maybe we could
>>>> change
>>>> > the names to be clearer.
>>>> Yes. Especially I want to change name *-proprietary into something like
>>>> *-extra because "proprietary".length() is long.
>>>>
>>>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>> other), to create *-proprietary directories is awkward, isn't it?
>>>> I know CONNECTORS-402 intent, but more simplified, can we say that "if
>>>> you use these jars,
>>>> please get source and run ''ant make-deps build"? -- I usually say so,
>>>> no problems occur for now.
>>>>
>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>
>>>> Ok.
>>>> But I want to know, can we deprecate example and move
>>>> example-proprietary to example?
>>>> I'd like to merge these examples. If proprietary jars exists, MCF just
>>>> have to load from extra-dir.
>>>> I assume that we can have extra-lib which is optional, If it exists,
>>>> MCF will load any Jars
>>>> found in this directory and use them to resolve connectors specified in
>>>> connectors.xml.
>>>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>>>
>>>> I just simplified directories since other Apache projects look like
>>>> much simpler.
>>>>
>>>> Regards,
>>>> Shinichiro Abe.
>>>>
>>>> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>>>>
>>>> > Hi Abe-san,
>>>> >
>>>> > Some comments:
>>>> >
>>>> > (1) I don't understand what you mean by, "because build.xml in
>>>> framework is
>>>> > currently too long to include class
>>>> > paths in lib". Can you clarify?
>>>> >
>>>> > (2) Some of these suggestions seem to be making distinctions between
>>>> files
>>>> > and directories that I don't understand the reason for. For example:
>>>> "in
>>>> > process-single directory we can
>>>> > create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
>>>> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars
>>>> in
>>>> > this way? They are all necessary at the root level to run the
>>>> example. I
>>>> > would not understand where to add a new jar with this arrangement
>>>> because
>>>> > the meaning of the directories is unclear.
>>>> >
>>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>>> other
>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>> under
>>>> > additional levels of hierarchy. I think it should be possible
>>>> immediately
>>>> > for a user to see what examples are available. But maybe we could
>>>> change
>>>> > the names to be clearer.
>>>> >
>>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>> >
>>>> > (5) Putting version numbers on jars is difficult in some cases,
>>>> especially
>>>> > in construction of start.jar, because the ant methods for constructing
>>>> > start.jar are poor. The version of each jar would need to be defined
>>>> > globally in the ant build, and included whenever the jar is
>>>> referenced.
>>>> >
>>>> > Karl
>>>>
>>>>
>>>
>>
>
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
I've moved CONNECTORS-345 to "Fix in 2.0" and attached your comments.
Let's talk about the jetty deployment issues in that ticket thread.
Karl
On Fri, Sep 26, 2014 at 5:16 AM, Karl Wright <da...@gmail.com> wrote:
> CONNECTORS-1048 created for including jar versions.
> Karl
>
> On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Abe-san,
>>
>> It seems to me that we are discussing some distinct and independent
>> improvements, so maybe I will open specific tickets for each one and we can
>> work on things one at a time. But, first:
>>
>> bq. I just simplified directories since other Apache projects look like
>> much simpler.
>>
>> That's true, but I don't know of any other Apache project which attempts
>> to handle conditional compilation to the degree that we have to in
>> ManifoldCF. So I think we should expect things to be more complicated. :-)
>>
>> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>> other), to create *-proprietary directories is awkward, isn't it?
>>
>> I've never particularly liked the -proprietary paradigm, but I could
>> think of no other way. The reasons behind it was that we *must* legally
>> exclude anything proprietary from the distribution. By making sure that
>> every proprietary jar is in a well-labeled directory, I can simply make
>> sure all such directories are excluded. But that also includes wars.
>> Since wars must include all jars, that meant that the build had to create
>> two different sets of wars, and put them in different directories. The
>> only other alternative is to guarantee that we always build our release
>> candidates from a fresh checkout each time, and that seemed risky.
>>
>> I agree that it would be more convenient for the user if our build
>> created one set of wars based on what was included by the user, and that we
>> shipped just the minimal binary that didn't include proprietary code. If
>> we can solve the war problem, we can do that.
>>
>> Karl
>>
>>
>>
>> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
>> shinichiro.abe.1@gmail.com> wrote:
>>
>>> Hi Karl,
>>>
>>> Thanks for the reply.
>>>
>>> > (1) I don't understand what you mean by, "because build.xml in
>>> framework is
>>> > currently too long to include class
>>> > paths in lib". Can you clarify?
>>>
>>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>>
>>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>>> If new jars introduced with new feature, manifest-cp-proprietary must be
>>> tweaked, too.
>>> (i18n properties files also bother us.)
>>>
>>> Alternatively, the ant task below is useful.
>>>
>>> common-build.xml: Add artifact-version to dest.
>>> + <get
>>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>>
>>> framework/build.xml: New manifest-cp property setting works fine.
>>>
>>> - <property name="manifest-cp-74" value="${manifest-cp-73}
>>> ../lib/xmlsec.jar"/>
>>> - <property name="manifest-cp-75" value="${manifest-cp-74}
>>> ../lib/opensaml.jar"/>
>>>
>>> + <property name="liblocation" location="../lib" />
>>> + <path id="lib-jars">
>>> + <fileset dir="${liblocation}" includes="*.jar"/>
>>> + </path>
>>> + <pathconvert property="manifest-cp" refid="lib-jars"
>>> targetos="unix" pathsep=" ">
>>> + <map from="${liblocation}" to="../lib"/>
>>> + <map from="\" to="/"/>
>>> + </pathconvert>
>>>
>>> + <!-- <property name="manifest-cp" value="${manifest-cp-75}"/>
>>> -->
>>>
>>> > (2) Some of these suggestions seem to be making distinctions between
>>> files
>>> > and directories that I don't understand the reason for.
>>>
>>> 1. I came to mind CONNECTORS-345.
>>> 2. Currently I have to create scripts in production:
>>> startup.sh --
>>> java -jar start.jar
>>> echo $! > mcf.pid
>>> shutdown.sh --
>>> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>>> agent-shutdown AFAIK but this script isn't cool.
>>>
>>> 3. start.jar command line options are useful.
>>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>>
>>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat. Of
>>> course combined.war can do that though.
>>> I thought additional jars could copy to another directory, and might
>>> load these using Custom ClassLoader.
>>>
>>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>>
>>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>>> requestHeaderSize and so on in jetty.xml --
>>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>>> I will drop these suggestions.
>>>
>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>> other
>>> > hierarchy additions seem less helpful, such as hiding the examples
>>> under
>>> > additional levels of hierarchy. I think it should be possible
>>> immediately
>>> > for a user to see what examples are available. But maybe we could
>>> change
>>> > the names to be clearer.
>>> Yes. Especially I want to change name *-proprietary into something like
>>> *-extra because "proprietary".length() is long.
>>>
>>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>> other), to create *-proprietary directories is awkward, isn't it?
>>> I know CONNECTORS-402 intent, but more simplified, can we say that "if
>>> you use these jars,
>>> please get source and run ''ant make-deps build"? -- I usually say so,
>>> no problems occur for now.
>>>
>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>> > directory and an "open" directory", with xxx and yyy under them.
>>>
>>> Ok.
>>> But I want to know, can we deprecate example and move
>>> example-proprietary to example?
>>> I'd like to merge these examples. If proprietary jars exists, MCF just
>>> have to load from extra-dir.
>>> I assume that we can have extra-lib which is optional, If it exists, MCF
>>> will load any Jars
>>> found in this directory and use them to resolve connectors specified in
>>> connectors.xml.
>>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>>
>>> I just simplified directories since other Apache projects look like much
>>> simpler.
>>>
>>> Regards,
>>> Shinichiro Abe.
>>>
>>> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>>>
>>> > Hi Abe-san,
>>> >
>>> > Some comments:
>>> >
>>> > (1) I don't understand what you mean by, "because build.xml in
>>> framework is
>>> > currently too long to include class
>>> > paths in lib". Can you clarify?
>>> >
>>> > (2) Some of these suggestions seem to be making distinctions between
>>> files
>>> > and directories that I don't understand the reason for. For example:
>>> "in
>>> > process-single directory we can
>>> > create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
>>> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars in
>>> > this way? They are all necessary at the root level to run the
>>> example. I
>>> > would not understand where to add a new jar with this arrangement
>>> because
>>> > the meaning of the directories is unclear.
>>> >
>>> > (3) I agree with "connector-specific-processes" and "plugins". But
>>> other
>>> > hierarchy additions seem less helpful, such as hiding the examples
>>> under
>>> > additional levels of hierarchy. I think it should be possible
>>> immediately
>>> > for a user to see what examples are available. But maybe we could
>>> change
>>> > the names to be clearer.
>>> >
>>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>>> > directory and an "open" directory", with xxx and yyy under them.
>>> >
>>> > (5) Putting version numbers on jars is difficult in some cases,
>>> especially
>>> > in construction of start.jar, because the ant methods for constructing
>>> > start.jar are poor. The version of each jar would need to be defined
>>> > globally in the ant build, and included whenever the jar is referenced.
>>> >
>>> > Karl
>>>
>>>
>>
>
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
CONNECTORS-1048 created for including jar versions.
Karl
On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Abe-san,
>
> It seems to me that we are discussing some distinct and independent
> improvements, so maybe I will open specific tickets for each one and we can
> work on things one at a time. But, first:
>
> bq. I just simplified directories since other Apache projects look like
> much simpler.
>
> That's true, but I don't know of any other Apache project which attempts
> to handle conditional compilation to the degree that we have to in
> ManifoldCF. So I think we should expect things to be more complicated. :-)
>
> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
> other), to create *-proprietary directories is awkward, isn't it?
>
> I've never particularly liked the -proprietary paradigm, but I could think
> of no other way. The reasons behind it was that we *must* legally exclude
> anything proprietary from the distribution. By making sure that every
> proprietary jar is in a well-labeled directory, I can simply make sure all
> such directories are excluded. But that also includes wars. Since wars
> must include all jars, that meant that the build had to create two
> different sets of wars, and put them in different directories. The only
> other alternative is to guarantee that we always build our release
> candidates from a fresh checkout each time, and that seemed risky.
>
> I agree that it would be more convenient for the user if our build created
> one set of wars based on what was included by the user, and that we shipped
> just the minimal binary that didn't include proprietary code. If we can
> solve the war problem, we can do that.
>
> Karl
>
>
>
> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
> shinichiro.abe.1@gmail.com> wrote:
>
>> Hi Karl,
>>
>> Thanks for the reply.
>>
>> > (1) I don't understand what you mean by, "because build.xml in
>> framework is
>> > currently too long to include class
>> > paths in lib". Can you clarify?
>>
>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>
>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>> If new jars introduced with new feature, manifest-cp-proprietary must be
>> tweaked, too.
>> (i18n properties files also bother us.)
>>
>> Alternatively, the ant task below is useful.
>>
>> common-build.xml: Add artifact-version to dest.
>> + <get
>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>
>> framework/build.xml: New manifest-cp property setting works fine.
>>
>> - <property name="manifest-cp-74" value="${manifest-cp-73}
>> ../lib/xmlsec.jar"/>
>> - <property name="manifest-cp-75" value="${manifest-cp-74}
>> ../lib/opensaml.jar"/>
>>
>> + <property name="liblocation" location="../lib" />
>> + <path id="lib-jars">
>> + <fileset dir="${liblocation}" includes="*.jar"/>
>> + </path>
>> + <pathconvert property="manifest-cp" refid="lib-jars"
>> targetos="unix" pathsep=" ">
>> + <map from="${liblocation}" to="../lib"/>
>> + <map from="\" to="/"/>
>> + </pathconvert>
>>
>> + <!-- <property name="manifest-cp" value="${manifest-cp-75}"/> -->
>>
>> > (2) Some of these suggestions seem to be making distinctions between
>> files
>> > and directories that I don't understand the reason for.
>>
>> 1. I came to mind CONNECTORS-345.
>> 2. Currently I have to create scripts in production:
>> startup.sh --
>> java -jar start.jar
>> echo $! > mcf.pid
>> shutdown.sh --
>> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>> agent-shutdown AFAIK but this script isn't cool.
>>
>> 3. start.jar command line options are useful.
>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>
>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat. Of
>> course combined.war can do that though.
>> I thought additional jars could copy to another directory, and might load
>> these using Custom ClassLoader.
>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>
>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>> requestHeaderSize and so on in jetty.xml --
>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>> I will drop these suggestions.
>>
>> > (3) I agree with "connector-specific-processes" and "plugins". But
>> other
>> > hierarchy additions seem less helpful, such as hiding the examples under
>> > additional levels of hierarchy. I think it should be possible
>> immediately
>> > for a user to see what examples are available. But maybe we could
>> change
>> > the names to be clearer.
>> Yes. Especially I want to change name *-proprietary into something like
>> *-extra because "proprietary".length() is long.
>>
>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>> other), to create *-proprietary directories is awkward, isn't it?
>> I know CONNECTORS-402 intent, but more simplified, can we say that "if
>> you use these jars,
>> please get source and run ''ant make-deps build"? -- I usually say so, no
>> problems occur for now.
>>
>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>> > directory and an "open" directory", with xxx and yyy under them.
>>
>> Ok.
>> But I want to know, can we deprecate example and move
>> example-proprietary to example?
>> I'd like to merge these examples. If proprietary jars exists, MCF just
>> have to load from extra-dir.
>> I assume that we can have extra-lib which is optional, If it exists, MCF
>> will load any Jars
>> found in this directory and use them to resolve connectors specified in
>> connectors.xml.
>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>
>> I just simplified directories since other Apache projects look like much
>> simpler.
>>
>> Regards,
>> Shinichiro Abe.
>>
>> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>>
>> > Hi Abe-san,
>> >
>> > Some comments:
>> >
>> > (1) I don't understand what you mean by, "because build.xml in
>> framework is
>> > currently too long to include class
>> > paths in lib". Can you clarify?
>> >
>> > (2) Some of these suggestions seem to be making distinctions between
>> files
>> > and directories that I don't understand the reason for. For example:
>> "in
>> > process-single directory we can
>> > create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
>> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars in
>> > this way? They are all necessary at the root level to run the
>> example. I
>> > would not understand where to add a new jar with this arrangement
>> because
>> > the meaning of the directories is unclear.
>> >
>> > (3) I agree with "connector-specific-processes" and "plugins". But
>> other
>> > hierarchy additions seem less helpful, such as hiding the examples under
>> > additional levels of hierarchy. I think it should be possible
>> immediately
>> > for a user to see what examples are available. But maybe we could
>> change
>> > the names to be clearer.
>> >
>> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
>> > yyy-proprietary, it would be more appropriate to have a "proprietary"
>> > directory and an "open" directory", with xxx and yyy under them.
>> >
>> > (5) Putting version numbers on jars is difficult in some cases,
>> especially
>> > in construction of start.jar, because the ant methods for constructing
>> > start.jar are poor. The version of each jar would need to be defined
>> > globally in the ant build, and included whenever the jar is referenced.
>> >
>> > Karl
>>
>>
>
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
Hi Abe-san,
It seems to me that we are discussing some distinct and independent
improvements, so maybe I will open specific tickets for each one and we can
work on things one at a time. But, first:
bq. I just simplified directories since other Apache projects look like
much simpler.
That's true, but I don't know of any other Apache project which attempts to
handle conditional compilation to the degree that we have to in
ManifoldCF. So I think we should expect things to be more complicated. :-)
bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
other), to create *-proprietary directories is awkward, isn't it?
I've never particularly liked the -proprietary paradigm, but I could think
of no other way. The reasons behind it was that we *must* legally exclude
anything proprietary from the distribution. By making sure that every
proprietary jar is in a well-labeled directory, I can simply make sure all
such directories are excluded. But that also includes wars. Since wars
must include all jars, that meant that the build had to create two
different sets of wars, and put them in different directories. The only
other alternative is to guarantee that we always build our release
candidates from a fresh checkout each time, and that seemed risky.
I agree that it would be more convenient for the user if our build created
one set of wars based on what was included by the user, and that we shipped
just the minimal binary that didn't include proprietary code. If we can
solve the war problem, we can do that.
Karl
On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <shinichiro.abe.1@gmail.com
> wrote:
> Hi Karl,
>
> Thanks for the reply.
>
> > (1) I don't understand what you mean by, "because build.xml in framework
> is
> > currently too long to include class
> > paths in lib". Can you clarify?
>
> Rerated to (5) , manifest-cp configuration is not cool at all.
>
> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
> If new jars introduced with new feature, manifest-cp-proprietary must be
> tweaked, too.
> (i18n properties files also bother us.)
>
> Alternatively, the ant task below is useful.
>
> common-build.xml: Add artifact-version to dest.
> + <get
> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>
> framework/build.xml: New manifest-cp property setting works fine.
>
> - <property name="manifest-cp-74" value="${manifest-cp-73}
> ../lib/xmlsec.jar"/>
> - <property name="manifest-cp-75" value="${manifest-cp-74}
> ../lib/opensaml.jar"/>
>
> + <property name="liblocation" location="../lib" />
> + <path id="lib-jars">
> + <fileset dir="${liblocation}" includes="*.jar"/>
> + </path>
> + <pathconvert property="manifest-cp" refid="lib-jars"
> targetos="unix" pathsep=" ">
> + <map from="${liblocation}" to="../lib"/>
> + <map from="\" to="/"/>
> + </pathconvert>
>
> + <!-- <property name="manifest-cp" value="${manifest-cp-75}"/> -->
>
> > (2) Some of these suggestions seem to be making distinctions between
> files
> > and directories that I don't understand the reason for.
>
> 1. I came to mind CONNECTORS-345.
> 2. Currently I have to create scripts in production:
> startup.sh --
> java -jar start.jar
> echo $! > mcf.pid
> shutdown.sh --
> kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
> agent-shutdown AFAIK but this script isn't cool.
>
> 3. start.jar command line options are useful.
> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>
> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat. Of
> course combined.war can do that though.
> I thought additional jars could copy to another directory, and might load
> these using Custom ClassLoader.
> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>
> 5. But If users don't want to config jetty.xml(e.g. thread pool,
> requestHeaderSize and so on in jetty.xml --
> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
> I will drop these suggestions.
>
> > (3) I agree with "connector-specific-processes" and "plugins". But other
> > hierarchy additions seem less helpful, such as hiding the examples under
> > additional levels of hierarchy. I think it should be possible
> immediately
> > for a user to see what examples are available. But maybe we could change
> > the names to be clearer.
> Yes. Especially I want to change name *-proprietary into something like
> *-extra because "proprietary".length() is long.
>
> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and other),
> to create *-proprietary directories is awkward, isn't it?
> I know CONNECTORS-402 intent, but more simplified, can we say that "if you
> use these jars,
> please get source and run ''ant make-deps build"? -- I usually say so, no
> problems occur for now.
>
> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
> > yyy-proprietary, it would be more appropriate to have a "proprietary"
> > directory and an "open" directory", with xxx and yyy under them.
>
> Ok.
> But I want to know, can we deprecate example and move example-proprietary
> to example?
> I'd like to merge these examples. If proprietary jars exists, MCF just
> have to load from extra-dir.
> I assume that we can have extra-lib which is optional, If it exists, MCF
> will load any Jars
> found in this directory and use them to resolve connectors specified in
> connectors.xml.
> (Solr has plugin class loader at $SOLR_HOME/lib)
>
> I just simplified directories since other Apache projects look like much
> simpler.
>
> Regards,
> Shinichiro Abe.
>
> On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
>
> > Hi Abe-san,
> >
> > Some comments:
> >
> > (1) I don't understand what you mean by, "because build.xml in framework
> is
> > currently too long to include class
> > paths in lib". Can you clarify?
> >
> > (2) Some of these suggestions seem to be making distinctions between
> files
> > and directories that I don't understand the reason for. For example: "in
> > process-single directory we can
> > create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
> > dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars in
> > this way? They are all necessary at the root level to run the example.
> I
> > would not understand where to add a new jar with this arrangement because
> > the meaning of the directories is unclear.
> >
> > (3) I agree with "connector-specific-processes" and "plugins". But other
> > hierarchy additions seem less helpful, such as hiding the examples under
> > additional levels of hierarchy. I think it should be possible
> immediately
> > for a user to see what examples are available. But maybe we could change
> > the names to be clearer.
> >
> > (4) Rather than group together xxx and xxx-proprietary, and yyy and
> > yyy-proprietary, it would be more appropriate to have a "proprietary"
> > directory and an "open" directory", with xxx and yyy under them.
> >
> > (5) Putting version numbers on jars is difficult in some cases,
> especially
> > in construction of start.jar, because the ant methods for constructing
> > start.jar are poor. The version of each jar would need to be defined
> > globally in the ant build, and included whenever the jar is referenced.
> >
> > Karl
>
>
Re: MCF 2.x binary package directory structure
Posted by Shinichiro Abe <sh...@gmail.com>.
Hi Karl,
Thanks for the reply.
> (1) I don't understand what you mean by, "because build.xml in framework is
> currently too long to include class
> paths in lib". Can you clarify?
Rerated to (5) , manifest-cp configuration is not cool at all.
https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
If new jars introduced with new feature, manifest-cp-proprietary must be tweaked, too.
(i18n properties files also bother us.)
Alternatively, the ant task below is useful.
common-build.xml: Add artifact-version to dest.
+ <get src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}" dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
framework/build.xml: New manifest-cp property setting works fine.
- <property name="manifest-cp-74" value="${manifest-cp-73} ../lib/xmlsec.jar"/>
- <property name="manifest-cp-75" value="${manifest-cp-74} ../lib/opensaml.jar"/>
+ <property name="liblocation" location="../lib" />
+ <path id="lib-jars">
+ <fileset dir="${liblocation}" includes="*.jar"/>
+ </path>
+ <pathconvert property="manifest-cp" refid="lib-jars" targetos="unix" pathsep=" ">
+ <map from="${liblocation}" to="../lib"/>
+ <map from="\" to="/"/>
+ </pathconvert>
+ <!-- <property name="manifest-cp" value="${manifest-cp-75}"/> -->
> (2) Some of these suggestions seem to be making distinctions between files
> and directories that I don't understand the reason for.
1. I came to mind CONNECTORS-345.
2. Currently I have to create scripts in production:
startup.sh --
java -jar start.jar
echo $! > mcf.pid
shutdown.sh --
kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call agent-shutdown AFAIK but this script isn't cool.
3. start.jar command line options are useful.
http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat. Of course combined.war can do that though.
I thought additional jars could copy to another directory, and might load these using Custom ClassLoader.
http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
5. But If users don't want to config jetty.xml(e.g. thread pool, requestHeaderSize and so on in jetty.xml --
http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
I will drop these suggestions.
> (3) I agree with "connector-specific-processes" and "plugins". But other
> hierarchy additions seem less helpful, such as hiding the examples under
> additional levels of hierarchy. I think it should be possible immediately
> for a user to see what examples are available. But maybe we could change
> the names to be clearer.
Yes. Especially I want to change name *-proprietary into something like *-extra because "proprietary".length() is long.
BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and other), to create *-proprietary directories is awkward, isn't it?
I know CONNECTORS-402 intent, but more simplified, can we say that "if you use these jars,
please get source and run ''ant make-deps build"? -- I usually say so, no problems occur for now.
> (4) Rather than group together xxx and xxx-proprietary, and yyy and
> yyy-proprietary, it would be more appropriate to have a "proprietary"
> directory and an "open" directory", with xxx and yyy under them.
Ok.
But I want to know, can we deprecate example and move example-proprietary to example?
I'd like to merge these examples. If proprietary jars exists, MCF just have to load from extra-dir.
I assume that we can have extra-lib which is optional, If it exists, MCF will load any Jars
found in this directory and use them to resolve connectors specified in connectors.xml.
(Solr has plugin class loader at $SOLR_HOME/lib)
I just simplified directories since other Apache projects look like much simpler.
Regards,
Shinichiro Abe.
On 2014/09/26, at 3:01, Karl Wright <da...@gmail.com> wrote:
> Hi Abe-san,
>
> Some comments:
>
> (1) I don't understand what you mean by, "because build.xml in framework is
> currently too long to include class
> paths in lib". Can you clarify?
>
> (2) Some of these suggestions seem to be making distinctions between files
> and directories that I don't understand the reason for. For example: "in
> process-single directory we can
> create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
> dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars in
> this way? They are all necessary at the root level to run the example. I
> would not understand where to add a new jar with this arrangement because
> the meaning of the directories is unclear.
>
> (3) I agree with "connector-specific-processes" and "plugins". But other
> hierarchy additions seem less helpful, such as hiding the examples under
> additional levels of hierarchy. I think it should be possible immediately
> for a user to see what examples are available. But maybe we could change
> the names to be clearer.
>
> (4) Rather than group together xxx and xxx-proprietary, and yyy and
> yyy-proprietary, it would be more appropriate to have a "proprietary"
> directory and an "open" directory", with xxx and yyy under them.
>
> (5) Putting version numbers on jars is difficult in some cases, especially
> in construction of start.jar, because the ant methods for constructing
> start.jar are poor. The version of each jar would need to be defined
> globally in the ant build, and included whenever the jar is referenced.
>
> Karl
Re: MCF 2.x binary package directory structure
Posted by Karl Wright <da...@gmail.com>.
Hi Abe-san,
Some comments:
(1) I don't understand what you mean by, "because build.xml in framework is
currently too long to include class
paths in lib". Can you clarify?
(2) Some of these suggestions seem to be making distinctions between files
and directories that I don't understand the reason for. For example: "in
process-single directory we can
create lib dir(for jetty jars), lib/ext dir(for logger jars), resource
dir(for logging.ini) and etc dir(for jetty.xml)." Why separate jars in
this way? They are all necessary at the root level to run the example. I
would not understand where to add a new jar with this arrangement because
the meaning of the directories is unclear.
(3) I agree with "connector-specific-processes" and "plugins". But other
hierarchy additions seem less helpful, such as hiding the examples under
additional levels of hierarchy. I think it should be possible immediately
for a user to see what examples are available. But maybe we could change
the names to be clearer.
(4) Rather than group together xxx and xxx-proprietary, and yyy and
yyy-proprietary, it would be more appropriate to have a "proprietary"
directory and an "open" directory", with xxx and yyy under them.
(5) Putting version numbers on jars is difficult in some cases, especially
in construction of start.jar, because the ant methods for constructing
start.jar are poor. The version of each jar would need to be defined
globally in the ant build, and included whenever the jar is referenced.
Karl