You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Paolo Castagna <ca...@googlemail.com> on 2011/11/17 00:42:03 UTC
Jena (i.e. jena-core): ARQ isn't included in the distribution zip
file
Hi,
I am not sure the Jena distribution zip is ready: ARQ is missing from it
and we have the old problem of circular dependencies between Jena and ARQ.
I tried fixing this doing this:
Index: pom.xml
===================================================================
--- pom.xml (revision 1202912)
+++ pom.xml (working copy)
@@ -33,11 +33,20 @@
</properties>
<dependencies>
+
<dependency>
- <groupId>org.apache.jena</groupId>
- <artifactId>jena-iri</artifactId>
- <version>0.9.0-incubating-SNAPSHOT</version>
+ <groupId>org.apache.jena</groupId>
+ <artifactId>jena-iri</artifactId>
+ <version>0.9.0-incubating-SNAPSHOT</version>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.jena</groupId>
+ <artifactId>jena-arq</artifactId>
+ <version>${ver.arq}</version>
+ <scope>runtime</scope>
+ </dependency>
+
</dependencies>
<build>
@@ -121,6 +130,31 @@
</plugin>
<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <configuration>
+ <outputDirectory>${project.basedir}/lib</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>true</overWriteSnapshots>
+ </configuration>
+ <executions>
+ <execution>
+ <id>copy-source-dependencies-for-assembly</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <classifier>sources</classifier>
+
<failOnMissingClassifierArtifact>false</failOnMissingClassifierArtifact>
+ <includeScope>runtime</includeScope>
+
<outputDirectory>${project.build.directory}/dependency-sources</outputDirectory>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
Index: assembly.xml
===================================================================
--- assembly.xml (revision 1202912)
+++ assembly.xml (working copy)
@@ -38,8 +38,7 @@
<directory>${project.basedir}/lib</directory>
<outputDirectory>lib</outputDirectory>
<includes>
- <include>arq-*</include>
- <include>lucene-*</include>
+ <include>jena-arq-*</include>
<include>stax-api-*</include>
<include>wstx-asl-*</include>
</includes>
@@ -50,7 +49,8 @@
<directory>${project.build.directory}/dependency-sources</directory>
<outputDirectory>lib-src</outputDirectory>
<includes>
- <include>iri-*-sources.jar</include>
+ <include>jena-iri-*-sources.jar</include>
+ <include>jena-arq-*-sources.jar</include>
</includes>
</fileSet>
@@ -58,8 +58,8 @@
<directory>${project.build.directory}</directory>
<outputDirectory>lib-src</outputDirectory>
<includes>
- <include>jena-${project.version}-sources.jar</include>
- <include>jena-${project.version}-tests-sources.jar</include>
+ <include>jena-core-${project.version}-sources.jar</include>
+ <include>jena-core-${project.version}-tests-sources.jar</include>
</includes>
</fileSet>
However, ARQ dependencies are not included transitively. This is a problem.
What about creating a jena-dist | jena-all module whose aim is just to create
the distribution? We discussed this already, but I am not sure this is included
in the first Apache release or not. This module can depend on Jena as well as
ARQ and therefore we can have an assembly.xml which includes all the necessary
dependencies.
If we agree on this, I'll provide an initial version tomorrow (just let me know
your preference for the name of the module).
If we do not agree on this, well we have a problem and at the moment I do not
have another proposal on how to solve this.
Paolo
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Andy
Andy Seaborne wrote:
> On 20/11/11 14:46, Paolo Castagna wrote:
>> I was just saying that a good practice it to use src/test/resources
>> and load data files necessary for tests via classpath so that it's
>> possible and trivial to ship a self contained and working test suite
>> as a single jar. Apply this to what we have has a big cost for little
>> or close to zero value. So, I aggree on no action on this.
>
> This is the web!
>
> Loading from the classpath does not work - some tests need a base URI
> and some test load files using relative URIs.
>
>>> 0/ Make a top level module of Jena2
>> JenaDist should have JenaTop as parent pom, but other than this I do not
>> see what else we would need.
>
> We need to have a proper SVN area for it. We can't go with something
> tucked under /Scratch/PC/
Sure.
I put it there since is was just an experiment to communicate the idea.
I am also not sure about the name: JenaDist? JenaAll? Others?
> Can you do that?
Sure.
I would call it JenaDist (and jena-dist the Maven name) and put it here:
https://svn.apache.org/repos/asf/incubator/jena/Jena2/JenaDist
Ok?
>>> 4/ Needs the change logs from Jena and ARQ at least.
>>> I don't how to mechanism this - copy into JenaDist for now.
>> Ok.
>>
>> JIRA has a "Change Log" tab which could help automating this (I often
>> forget to update the CHANGELOG.txt files and I agree that is important).
>
> Can we just concentrate on getting release done and not adding work to
> the timeline? We have the ChangeLogs already (and the JIRA log is
> somewhat minimal).
Ack.
> We are not trying to detail every change (too much) - we're trying to
> tell people what they need to know, which isn't a detail dump of
> everything.
>
>> A final comment. We will probably not get everything right with the first
>> release, that's fine so long we are ready to quickly fix (via a bug fix
>> release) problems as we discover them. I think "release early and often"
>> applies here as we learn the release process in Apache and we try to fit
>> what we used to do in a different context (with different constraints).
>
> I don't see it like that; it's not about the code, it's the process.
>
> We are trying to get through Apache process, including checking on
> incubator-general@ It's useful to read other incubator projects getting
> through that.
>
> We will only get some much bandwidth from people there; it's a busy place.
>
> While we can do dev builds, there isn't "quick" for releases. My idea
> of "quick" is a few hours; loop time here is going to one week+ for a
> two sequential votes. Getting it right is quite a good idea.
Ok.
Paolo
>
> Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Andy Seaborne <an...@apache.org>.
On 20/11/11 14:46, Paolo Castagna wrote:
> I was just saying that a good practice it to use src/test/resources
> and load data files necessary for tests via classpath so that it's
> possible and trivial to ship a self contained and working test suite
> as a single jar. Apply this to what we have has a big cost for little
> or close to zero value. So, I aggree on no action on this.
This is the web!
Loading from the classpath does not work - some tests need a base URI
and some test load files using relative URIs.
>> 0/ Make a top level module of Jena2
> JenaDist should have JenaTop as parent pom, but other than this I do not
> see what else we would need.
We need to have a proper SVN area for it. We can't go with something
tucked under /Scratch/PC/
Can you do that?
>> 4/ Needs the change logs from Jena and ARQ at least.
>> I don't how to mechanism this - copy into JenaDist for now.
> Ok.
>
> JIRA has a "Change Log" tab which could help automating this (I often
> forget to update the CHANGELOG.txt files and I agree that is important).
Can we just concentrate on getting release done and not adding work to
the timeline? We have the ChangeLogs already (and the JIRA log is
somewhat minimal).
We are not trying to detail every change (too much) - we're trying to
tell people what they need to know, which isn't a detail dump of everything.
> A final comment. We will probably not get everything right with the first
> release, that's fine so long we are ready to quickly fix (via a bug fix
> release) problems as we discover them. I think "release early and often"
> applies here as we learn the release process in Apache and we try to fit
> what we used to do in a different context (with different constraints).
I don't see it like that; it's not about the code, it's the process.
We are trying to get through Apache process, including checking on
incubator-general@ It's useful to read other incubator projects getting
through that.
We will only get some much bandwidth from people there; it's a busy place.
While we can do dev builds, there isn't "quick" for releases. My idea
of "quick" is a few hours; loop time here is going to one week+ for a
two sequential votes. Getting it right is quite a good idea.
Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Andy
Andy Seaborne wrote:
> On 17/11/11 13:22, Paolo Castagna wrote:
>> Hi Andy
>>
>> Andy Seaborne wrote:
>>> On 17/11/11 00:57, Paolo Castagna wrote:
>>>> Paolo Castagna wrote:
>>>>> If we agree on this, I'll provide an initial version tomorrow (just
>>>>> let me know your preference for the name of the module).
>>>>
>>>> Just to make things more explicit, here is an example (which is still
>>>> missing
>>>> a lot of things, but it has all the necessary jar files):
>>>> http://svn.apache.org/repos/asf/incubator/jena/Scratch/PC/JenaDist/trunk/
>>>>
>>>>
>>>>
>>>> Paolo
>>>
>>> If we do this, we might as well drop the zips for the other packages.
>>
>> I think it would be a good idea to drop the zip files and just have one
>> zip for the Apache Jena distribution (including: jena-core, jena-arq,
>> jena-tdb (why not?), jena-iri, jena-larq as well as all their transitive
>> dependencies).
>
> TDB needs work (e.g. JENA-143). Fuseki isn't ready.
>
> So let us aim for IRI/core/ARQ/LARQ.
Ack.
>> This would reduce the cost of maintaining separate assembly files.
>>
>> Also, I need to investigate the use of moduleSets in the assembly.xml
>> descriptor... it might be better if we want to aggregate sources (and/or
>> shell scripts, no idea if this is possible).
>
> Progress? Is it needed?
None (I'll look at it on Monday). Not needed.
However, I have no idea how to aggregate sources from different modules
(this is needed as we need to publish the sources as well as binaries).
>>> And the testing assemblies are not very helpful.
>>
>> I am not saying we do this, however an option is to put testing files
>> in the src/test/resources directory and let Maven include them in the
>> -test.jar. However, tests need to load files via classloader instead of
>> directly from the file system. Maybe this is something to aim in the
>> future if there is agreement. Certainly not now.
>
> Forget testing material in the distribution. It's not a helpful as it
> might seem as you need all the "test" artifact jars and the test
> scripting areas. In the interests of time, I would not consider doing
> it this time round. Also, running the tests requires running from
> particular locations.
Ok.
I was just saying that a good practice it to use src/test/resources
and load data files necessary for tests via classpath so that it's
possible and trivial to ship a self contained and working test suite
as a single jar. Apply this to what we have has a big cost for little
or close to zero value. So, I aggree on no action on this.
>
>> I see some value in allowing people to depend on the test suite of
>> a project, it's not something that happens often, but sometimes it's
>> useful to depend/extend/run existing tests from a different module.
>>
>>> The only issue I see is time (but that looks OK) and risk (hard to
>>> tell).
>>
>> Yep, I don't disagree.
>>
>> Also, shell commands (from Jena, ARQ and TDB) probably should be moved on
>> a JenaDist module if we do that.
>
> Yes
>
>>> A crude solution is ship the ARQ zip as "apache-jena-VER.zip" but if
>>> there is time, then a distribution project is a step in the right
>>> direction.
>>
>> I like this idea, it could be ARQ zip or indeed TDB zip (even better).
>> But, it would be good to have all the Jena, ARQ and TDB command line
>> in it.
>
> In which case we have to solve the cross linkages so we might as well
> use JenaDist.
jena-core does not currently depend on jena-arq.
jena-arq depends on jena-core.
That's fine, no circular dependency.
We can use an assembly.xml in jena-arq to build a distribution including
jena-core, jena-iri, jena-arq. However, LARQ will not fit nicely with this
approach (since jena-arq does not depend on jena-larq).
Three options:
1/ We do this in LARQ
2/ We do this in JenaDist
3/ We do not include LARQ in the Jena distribution
2/ JenaDist is better IMHO.
3/ is also fine.
>
>> Maybe we should do this as a first step and move the assembly part to
>> a JenaDist later when we do a second or subsequent release.
>
> I scanned JenaDist and if we are going to use this time round:
Oh, yeah, as I said: it is missing a lot of things. I wanted to share it
to show the idea of having a module with the dependencies on all the things
we want to include and no code whose aim is just to build the distribution.
> 0/ Make a top level module of Jena2
JenaDist should have JenaTop as parent pom, but other than this I do not
see what else we would need.
> 1/ A README needs to be written (inc how-to-use instructions)
> The zip needs to be moderately self contained.
> Explain what this is.
> Brief how to use "put all the jars on the classpath"
> Where to get javadoc.
> The current pointer to the website for a standalone zip is a bit harsh.
> Setting environment variables.
Yes.
> 2/ Needs to include bin/ and bat/
> I don't how to mechanism this - copy into JenaDist for now.
> README or etc needs something on setting environment variables.
Yes.
> 3/ NOTICE is wrong (old text)
Easy to fix.
Is this the NOTICE we want?
"""
Apache Jena - {module} module
Copyright 2011 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
"""
>
> 4/ Needs the change logs from Jena and ARQ at least.
> I don't how to mechanism this - copy into JenaDist for now.
Ok.
JIRA has a "Change Log" tab which could help automating this (I often
forget to update the CHANGELOG.txt files and I agree that is important).
>
> 5/ How do we have a "Apache source distribution"?
> I think this is going to be interesting to explain to
> incubator-general.
This is one of the reasons why I wanted to explore the moduleSets option.
I think that could be the way to achieve this, but I am not sure until
I do it.
> Are there any blockers?
The source distribution is a MUST, so until we do that, that is a blocker.
Other than this, I do not see blockers or problems.
A final comment. We will probably not get everything right with the first
release, that's fine so long we are ready to quickly fix (via a bug fix
release) problems as we discover them. I think "release early and often"
applies here as we learn the release process in Apache and we try to fit
what we used to do in a different context (with different constraints).
Paolo
>
> Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Andy Seaborne <an...@apache.org>.
On 17/11/11 13:22, Paolo Castagna wrote:
> Hi Andy
>
> Andy Seaborne wrote:
>> On 17/11/11 00:57, Paolo Castagna wrote:
>>> Paolo Castagna wrote:
>>>> If we agree on this, I'll provide an initial version tomorrow (just
>>>> let me know your preference for the name of the module).
>>>
>>> Just to make things more explicit, here is an example (which is still
>>> missing
>>> a lot of things, but it has all the necessary jar files):
>>> http://svn.apache.org/repos/asf/incubator/jena/Scratch/PC/JenaDist/trunk/
>>>
>>>
>>> Paolo
>>
>> If we do this, we might as well drop the zips for the other packages.
>
> I think it would be a good idea to drop the zip files and just have one
> zip for the Apache Jena distribution (including: jena-core, jena-arq,
> jena-tdb (why not?), jena-iri, jena-larq as well as all their transitive
> dependencies).
TDB needs work (e.g. JENA-143). Fuseki isn't ready.
So let us aim for IRI/core/ARQ/LARQ.
> This would reduce the cost of maintaining separate assembly files.
>
> Also, I need to investigate the use of moduleSets in the assembly.xml
> descriptor... it might be better if we want to aggregate sources (and/or
> shell scripts, no idea if this is possible).
Progress? Is it needed?
>> And the testing assemblies are not very helpful.
>
> I am not saying we do this, however an option is to put testing files
> in the src/test/resources directory and let Maven include them in the
> -test.jar. However, tests need to load files via classloader instead of
> directly from the file system. Maybe this is something to aim in the
> future if there is agreement. Certainly not now.
Forget testing material in the distribution. It's not a helpful as it
might seem as you need all the "test" artifact jars and the test
scripting areas. In the interests of time, I would not consider doing
it this time round. Also, running the tests requires running from
particular locations.
> I see some value in allowing people to depend on the test suite of
> a project, it's not something that happens often, but sometimes it's
> useful to depend/extend/run existing tests from a different module.
>
>> The only issue I see is time (but that looks OK) and risk (hard to tell).
>
> Yep, I don't disagree.
>
> Also, shell commands (from Jena, ARQ and TDB) probably should be moved on
> a JenaDist module if we do that.
Yes
>> A crude solution is ship the ARQ zip as "apache-jena-VER.zip" but if
>> there is time, then a distribution project is a step in the right
>> direction.
>
> I like this idea, it could be ARQ zip or indeed TDB zip (even better).
> But, it would be good to have all the Jena, ARQ and TDB command line
> in it.
In which case we have to solve the cross linkages so we might as well
use JenaDist.
> Maybe we should do this as a first step and move the assembly part to
> a JenaDist later when we do a second or subsequent release.
I scanned JenaDist and if we are going to use this time round:
0/ Make a top level module of Jena2
1/ A README needs to be written (inc how-to-use instructions)
The zip needs to be moderately self contained.
Explain what this is.
Brief how to use "put all the jars on the classpath"
Where to get javadoc.
The current pointer to the website for a standalone zip is a bit harsh.
Setting environment variables.
2/ Needs to include bin/ and bat/
I don't how to mechanism this - copy into JenaDist for now.
README or etc needs something on setting environment variables.
3/ NOTICE is wrong (old text)
4/ Needs the change logs from Jena and ARQ at least.
I don't how to mechanism this - copy into JenaDist for now.
5/ How do we have a "Apache source distribution"?
I think this is going to be interesting to explain to
incubator-general.
Are there any blockers?
Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Andy
Andy Seaborne wrote:
> On 17/11/11 00:57, Paolo Castagna wrote:
>> Paolo Castagna wrote:
>>> If we agree on this, I'll provide an initial version tomorrow (just
>>> let me know your preference for the name of the module).
>>
>> Just to make things more explicit, here is an example (which is still
>> missing
>> a lot of things, but it has all the necessary jar files):
>> http://svn.apache.org/repos/asf/incubator/jena/Scratch/PC/JenaDist/trunk/
>>
>> Paolo
>
> If we do this, we might as well drop the zips for the other packages.
I think it would be a good idea to drop the zip files and just have one
zip for the Apache Jena distribution (including: jena-core, jena-arq,
jena-tdb (why not?), jena-iri, jena-larq as well as all their transitive
dependencies).
This would reduce the cost of maintaining separate assembly files.
Also, I need to investigate the use of moduleSets in the assembly.xml
descriptor... it might be better if we want to aggregate sources (and/or
shell scripts, no idea if this is possible).
> And the testing assemblies are not very helpful.
I am not saying we do this, however an option is to put testing files
in the src/test/resources directory and let Maven include them in the
-test.jar. However, tests need to load files via classloader instead of
directly from the file system. Maybe this is something to aim in the
future if there is agreement. Certainly not now.
I see some value in allowing people to depend on the test suite of
a project, it's not something that happens often, but sometimes it's
useful to depend/extend/run existing tests from a different module.
> The only issue I see is time (but that looks OK) and risk (hard to tell).
Yep, I don't disagree.
Also, shell commands (from Jena, ARQ and TDB) probably should be moved on
a JenaDist module if we do that.
> A crude solution is ship the ARQ zip as "apache-jena-VER.zip" but if
> there is time, then a distribution project is a step in the right
> direction.
I like this idea, it could be ARQ zip or indeed TDB zip (even better).
But, it would be good to have all the Jena, ARQ and TDB command line
in it.
Maybe we should do this as a first step and move the assembly part to
a JenaDist later when we do a second or subsequent release.
Paolo
>
> Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Andy Seaborne <an...@apache.org>.
On 17/11/11 00:57, Paolo Castagna wrote:
> Paolo Castagna wrote:
>> If we agree on this, I'll provide an initial version tomorrow (just
>> let me know your preference for the name of the module).
>
> Just to make things more explicit, here is an example (which is still
> missing
> a lot of things, but it has all the necessary jar files):
> http://svn.apache.org/repos/asf/incubator/jena/Scratch/PC/JenaDist/trunk/
>
> Paolo
If we do this, we might as well drop the zips for the other packages.
And the testing assemblies are not very helpful.
The only issue I see is time (but that looks OK) and risk (hard to tell).
A crude solution is ship the ARQ zip as "apache-jena-VER.zip" but if
there is time, then a distribution project is a step in the right direction.
Andy
Re: Jena (i.e. jena-core): ARQ isn't included in the distribution
zip file
Posted by Paolo Castagna <ca...@googlemail.com>.
Paolo Castagna wrote:
> If we agree on this, I'll provide an initial version tomorrow (just let
> me know your preference for the name of the module).
Just to make things more explicit, here is an example (which is still missing
a lot of things, but it has all the necessary jar files):
http://svn.apache.org/repos/asf/incubator/jena/Scratch/PC/JenaDist/trunk/
Paolo