You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@archiva.apache.org by Johannes Utzig <ju...@googlemail.com> on 2011/08/17 21:53:35 UTC
using archiva as a p2 repository
Hi,
I don't know if any of you work with OSGi in general and eclipse plugins
in specific, but both technologies have gained a lot of momentum in
recent years.
Now with maven tycho finally available, OSGi, eclipse and maven are
really starting to grow together more closely.
Without getting into details, tycho resolves dependencies from both
maven repositories and p2 repositories and allows you to easily deploy
your OSGi bundles into a standard maven repository.
This works great at build time, but as of now, not so great at
development time. When developing OSGi bundles in eclipse, you typically
define a 'Target Platform' that contains all your dependencies and
eclipse uses this to automatically construct the classpath (very similar
to what m2eclipse does in a typical maven project).
Most of the time these Target Platform definitions consist of a list of
remote p2 repositories and eclipse downloads the required bundles
automatically on demand, again, very similar to the way it works with
maven/m2eclipse.
The problem is, eclipse only understands p2 repositories, not maven
repositories. So if for example you want to compile your projects at
development time against the OSGi bundles that are available in your
maven repository (deployed by yourself with maven tycho, or from an
external source like Spring Enterprise Repository), you simply cannot do
it.
First you need to either
-download all files manually and add them to your target platform
-or create a p2 repository from all the bundles and put that somewhere
on a webserver
None of the alternatives sound great to me. We already have an archiva
running that contains all the bundles we need, so I'd rather not
duplicate all the jars and put them somewhere else.
So I had the idea of using the archiva consumer API to crawl an archiva
repository and create the p2 repository files (artifacts.jar and
content.jar) on-the-fly. Sort of like a p2 view on a maven repository.
To my understanding Nexus Professional offers such a feature, but there
is no open source equivalent to this.
Over the last few days I was working on this idea a little and actually
have a working prototype now.
Whenever a new artifact gets deployed in archiva, the bundle manifest
(or eclipse feature.xml) is parsed and the information is appended to
the content.jar and artifacts.jar that I create in the root folder of
the archiva repository. That way you can use the archiva repository as a
regular p2 repository/eclipse update site and install everything that's
either an OSGi bundle, or an eclipse feature into your host eclipse or
your target platform.
Sorry for the long background story, but now finally to my questions.
I'd greatly appreciate if somebody could help me out with these:
1. I think this would be a very helpful feature for users of tycho and
archiva and I'd like to make this (once finished) available as open
source. I was wondering if this is something you'd be interested in
hosting directly as an apache archiva component
2. I could not find a way in the consumer API to determine if the
'Process All Artifacts' checkbox in 'admin/repositories.action' was
activated before the scan got triggered. A full scan is the perfect time
to throw away the old artifacts.jar and content.jar, but if the checkbox
is not activated, I end up with an incomplete repository. At the moment
I have overridden isProcessUnmodified() and return true. That works, but
is unfortunately very very expensive for this kind of consumer if the
repository is reasonably large.
3. I also could not find a way to get an event when a file is deleted.
How can a consumer find out when an artifact gets deleted? Especially
with snapshots being always unique since maven 3, the p2 files will
quickly explode when old snapshot entries don't get deleted regulary...
Thanks and best regards,
Johannes Utzig
Re: using archiva as a p2 repository
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Johannes,
FYI, we are working on the support of OBR in Archiva.
A POC has started in Karaf Cave, and tomorrow, I will start to work on
integrating Cave OBR into Archiva.
Regards
JB
On 08/17/2011 09:53 PM, Johannes Utzig wrote:
> Hi,
>
> I don't know if any of you work with OSGi in general and eclipse plugins
> in specific, but both technologies have gained a lot of momentum in
> recent years.
> Now with maven tycho finally available, OSGi, eclipse and maven are
> really starting to grow together more closely.
> Without getting into details, tycho resolves dependencies from both
> maven repositories and p2 repositories and allows you to easily deploy
> your OSGi bundles into a standard maven repository.
> This works great at build time, but as of now, not so great at
> development time. When developing OSGi bundles in eclipse, you typically
> define a 'Target Platform' that contains all your dependencies and
> eclipse uses this to automatically construct the classpath (very similar
> to what m2eclipse does in a typical maven project).
> Most of the time these Target Platform definitions consist of a list of
> remote p2 repositories and eclipse downloads the required bundles
> automatically on demand, again, very similar to the way it works with
> maven/m2eclipse.
> The problem is, eclipse only understands p2 repositories, not maven
> repositories. So if for example you want to compile your projects at
> development time against the OSGi bundles that are available in your
> maven repository (deployed by yourself with maven tycho, or from an
> external source like Spring Enterprise Repository), you simply cannot do
> it.
> First you need to either
> -download all files manually and add them to your target platform
> -or create a p2 repository from all the bundles and put that somewhere
> on a webserver
>
> None of the alternatives sound great to me. We already have an archiva
> running that contains all the bundles we need, so I'd rather not
> duplicate all the jars and put them somewhere else.
> So I had the idea of using the archiva consumer API to crawl an archiva
> repository and create the p2 repository files (artifacts.jar and
> content.jar) on-the-fly. Sort of like a p2 view on a maven repository.
> To my understanding Nexus Professional offers such a feature, but there
> is no open source equivalent to this.
>
> Over the last few days I was working on this idea a little and actually
> have a working prototype now.
> Whenever a new artifact gets deployed in archiva, the bundle manifest
> (or eclipse feature.xml) is parsed and the information is appended to
> the content.jar and artifacts.jar that I create in the root folder of
> the archiva repository. That way you can use the archiva repository as a
> regular p2 repository/eclipse update site and install everything that's
> either an OSGi bundle, or an eclipse feature into your host eclipse or
> your target platform.
>
> Sorry for the long background story, but now finally to my questions.
> I'd greatly appreciate if somebody could help me out with these:
> 1. I think this would be a very helpful feature for users of tycho and
> archiva and I'd like to make this (once finished) available as open
> source. I was wondering if this is something you'd be interested in
> hosting directly as an apache archiva component
>
> 2. I could not find a way in the consumer API to determine if the
> 'Process All Artifacts' checkbox in 'admin/repositories.action' was
> activated before the scan got triggered. A full scan is the perfect time
> to throw away the old artifacts.jar and content.jar, but if the checkbox
> is not activated, I end up with an incomplete repository. At the moment
> I have overridden isProcessUnmodified() and return true. That works, but
> is unfortunately very very expensive for this kind of consumer if the
> repository is reasonably large.
>
> 3. I also could not find a way to get an event when a file is deleted.
> How can a consumer find out when an artifact gets deleted? Especially
> with snapshots being always unique since maven 3, the p2 files will
> quickly explode when old snapshot entries don't get deleted regulary...
>
> Thanks and best regards,
> Johannes Utzig
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: using archiva as a p2 repository
Posted by Johannes Utzig <ju...@googlemail.com>.
Hi Oliver and Jean-Baptiste,
you guys are fast as lightning :-)
This is just terrific news. To my understanding the OBR format and the
p2 format are very very close to each other.
So if there is already something planned for OBR, it would just come
down to transforming the OBR information into a p2 compatible form and
additionally parsing eclipse feature jars.
When do you expect that something will be available in that area so that
I can try to adapt what I already have to the OBR implementation? Is
there anything that I can do to help? I don't have a lot of spare time
for this but I'd really love to see this functionality in archiva so I'm
willing to help to the best of my abilities.
So far I have a consumer that hooks into archiva, a reverse engineered
XSD and EMF model for the p2 XML and a hand written parser for OSGi
Manifests and feature jars that I was planning to replace with an
existing implementation later.
Not much, but it's a working POC and I'd figured I'd drop a note here
before investing more time in this (good thing I did).
Are these the JIRAs for it?
http://jira.codehaus.org/browse/MRM-1427
http://jira.codehaus.org/browse/MINDEXER-36
Sounds about perfect to me once it is OBR + p2 :-)
-Johannes
Am 08/17/2011 10:15 PM, schrieb Olivier Lamy:
> Hello Johannes ,
> All contribs are welcome !
>
> Note support of indexing osgi metadatas is now implemented in maven-indexer.
> So basically some osgi are coming here:-)
Am 08/17/2011 10:09 PM, schrieb Jean-Baptiste Onofré:
> FYI, we are working on the support of OBR in Archiva.
> A POC has started in Karaf Cave, and tomorrow, I will start to work on
> integrating Cave OBR into Archiva.
Re: using archiva as a p2 repository
Posted by Olivier Lamy <ol...@apache.org>.
Hello Johannes ,
All contribs are welcome !
Note support of indexing osgi metadatas is now implemented in maven-indexer.
So basically some osgi are coming here :-)
2011/8/17 Johannes Utzig <ju...@googlemail.com>:
> Hi,
>
> I don't know if any of you work with OSGi in general and eclipse plugins in
> specific, but both technologies have gained a lot of momentum in recent
> years.
> Now with maven tycho finally available, OSGi, eclipse and maven are really
> starting to grow together more closely.
> Without getting into details, tycho resolves dependencies from both maven
> repositories and p2 repositories and allows you to easily deploy your OSGi
> bundles into a standard maven repository.
> This works great at build time, but as of now, not so great at development
> time. When developing OSGi bundles in eclipse, you typically define a
> 'Target Platform' that contains all your dependencies and eclipse uses this
> to automatically construct the classpath (very similar to what m2eclipse
> does in a typical maven project).
> Most of the time these Target Platform definitions consist of a list of
> remote p2 repositories and eclipse downloads the required bundles
> automatically on demand, again, very similar to the way it works with
> maven/m2eclipse.
> The problem is, eclipse only understands p2 repositories, not maven
> repositories. So if for example you want to compile your projects at
> development time against the OSGi bundles that are available in your maven
> repository (deployed by yourself with maven tycho, or from an external
> source like Spring Enterprise Repository), you simply cannot do it.
> First you need to either
> -download all files manually and add them to your target platform
> -or create a p2 repository from all the bundles and put that somewhere on a
> webserver
>
> None of the alternatives sound great to me. We already have an archiva
> running that contains all the bundles we need, so I'd rather not duplicate
> all the jars and put them somewhere else.
> So I had the idea of using the archiva consumer API to crawl an archiva
> repository and create the p2 repository files (artifacts.jar and
> content.jar) on-the-fly. Sort of like a p2 view on a maven repository.
> To my understanding Nexus Professional offers such a feature, but there is
> no open source equivalent to this.
>
> Over the last few days I was working on this idea a little and actually have
> a working prototype now.
> Whenever a new artifact gets deployed in archiva, the bundle manifest (or
> eclipse feature.xml) is parsed and the information is appended to the
> content.jar and artifacts.jar that I create in the root folder of the
> archiva repository. That way you can use the archiva repository as a regular
> p2 repository/eclipse update site and install everything that's either an
> OSGi bundle, or an eclipse feature into your host eclipse or your target
> platform.
>
> Sorry for the long background story, but now finally to my questions. I'd
> greatly appreciate if somebody could help me out with these:
> 1. I think this would be a very helpful feature for users of tycho and
> archiva and I'd like to make this (once finished) available as open source.
> I was wondering if this is something you'd be interested in hosting directly
> as an apache archiva component
>
> 2. I could not find a way in the consumer API to determine if the 'Process
> All Artifacts' checkbox in 'admin/repositories.action' was activated before
> the scan got triggered. A full scan is the perfect time to throw away the
> old artifacts.jar and content.jar, but if the checkbox is not activated, I
> end up with an incomplete repository. At the moment I have overridden
> isProcessUnmodified() and return true. That works, but is unfortunately very
> very expensive for this kind of consumer if the repository is reasonably
> large.
>
> 3. I also could not find a way to get an event when a file is deleted. How
> can a consumer find out when an artifact gets deleted? Especially with
> snapshots being always unique since maven 3, the p2 files will quickly
> explode when old snapshot entries don't get deleted regulary...
>
> Thanks and best regards,
> Johannes Utzig
>
--
Olivier Lamy
Talend : http://talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: using archiva as a p2 repository
Posted by Olivier Lamy <ol...@apache.org>.
Hello,
2011/8/18 Johannes Utzig <ju...@googlemail.com>:
> Hi Brett,
>
> thank you very much for your fast reply.
> Please see comments inline
>
> Am 08/17/2011 10:21 PM, schrieb Brett Porter:
>>
>>> 1. I think this would be a very helpful feature for users of tycho and
>>> archiva and I'd like to make this (once finished) available as open source.
>>> I was wondering if this is something you'd be interested in hosting directly
>>> as an apache archive component
>>
>> Yes!
>>
>> Is it against 1.3.x or trunk?
>>
> I used 1.3.5 but it's only a prototype so far, should be easy to switch to
> trunk
IMHO you must work with trunk. We are more working on trunk than on branches
Regarding obr etc, yup you have a look at the good jira entries.
If you have any ideas do not hesitate to provide a patch :-)
>
>> If I understand correctly, it might be better to add a hook to allow the
>> custom code to remove the old files at the start of the scan when that is
>> checked, and not otherwise - rather than putting it in the consumer API
>> itself?
>
> Yes, that makes sense. I think I fell for the 'boolean executeOnEntireRepo'
> parameter, which sounded to me like it would correlate to the 'Process All
> Artifacts' flag, but as it turns out, it doesn't :-)
>
>> Implementors of
>> archiva-modules/archiva-base/archiva-repository-layer/src/main/java/org/apache/maven/archiva/repository/events/RepositoryListener.java
>>
>> (though it may not be consistently used for some deletions).
>>
>
> Exactly what I was looking for and kept missing for some reason. Thanks a
> lot for your help.
>
> -Johannes
>
--
Olivier Lamy
Talend : http://talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: using archiva as a p2 repository
Posted by Johannes Utzig <ju...@googlemail.com>.
Hi Brett,
thank you very much for your fast reply.
Please see comments inline
Am 08/17/2011 10:21 PM, schrieb Brett Porter:
>
>> 1. I think this would be a very helpful feature for users of tycho and archiva and I'd like to make this (once finished) available as open source. I was wondering if this is something you'd be interested in hosting directly as an apache archive component
> Yes!
>
> Is it against 1.3.x or trunk?
>
I used 1.3.5 but it's only a prototype so far, should be easy to switch
to trunk
> If I understand correctly, it might be better to add a hook to allow the custom code to remove the old files at the start of the scan when that is checked, and not otherwise - rather than putting it in the consumer API itself?
Yes, that makes sense. I think I fell for the 'boolean
executeOnEntireRepo' parameter, which sounded to me like it would
correlate to the 'Process All Artifacts' flag, but as it turns out, it
doesn't :-)
> Implementors of archiva-modules/archiva-base/archiva-repository-layer/src/main/java/org/apache/maven/archiva/repository/events/RepositoryListener.java
>
> (though it may not be consistently used for some deletions).
>
Exactly what I was looking for and kept missing for some reason. Thanks
a lot for your help.
-Johannes
Re: using archiva as a p2 repository
Posted by Brett Porter <br...@apache.org>.
On 18/08/2011, at 5:53 AM, Johannes Utzig wrote:
> Sorry for the long background story, but now finally to my questions. I'd greatly appreciate if somebody could help me out with these:
> 1. I think this would be a very helpful feature for users of tycho and archiva and I'd like to make this (once finished) available as open source. I was wondering if this is something you'd be interested in hosting directly as an apache archive component
Yes!
Is it against 1.3.x or trunk?
>
> 2. I could not find a way in the consumer API to determine if the 'Process All Artifacts' checkbox in 'admin/repositories.action' was activated before the scan got triggered. A full scan is the perfect time to throw away the old artifacts.jar and content.jar, but if the checkbox is not activated, I end up with an incomplete repository. At the moment I have overridden isProcessUnmodified() and return true. That works, but is unfortunately very very expensive for this kind of consumer if the repository is reasonably large.
If I understand correctly, it might be better to add a hook to allow the custom code to remove the old files at the start of the scan when that is checked, and not otherwise - rather than putting it in the consumer API itself?
>
> 3. I also could not find a way to get an event when a file is deleted. How can a consumer find out when an artifact gets deleted? Especially with snapshots being always unique since maven 3, the p2 files will quickly explode when old snapshot entries don't get deleted regulary…
Implementors of archiva-modules/archiva-base/archiva-repository-layer/src/main/java/org/apache/maven/archiva/repository/events/RepositoryListener.java
(though it may not be consistently used for some deletions).
- Brett
--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter