Re: Advice for managing large-scale OSGI repository for data analysis

[Nicolas Lalevée <ni...@hibnet.org>]

Le 30 août 2011 à 11:39, Nicolas Lalevée a écrit :

> 
> Le 30 août 2011 à 08:19, Marcel Bruch a écrit :
> 
>> Hi,
>> 
>> On 29.08.2011, at 23:25, Nicolas Lalevée wrote:
>> 
>>>> 1. Is it supported to copy/mirror an Eclipse P2 repository into a local Ivy repository?
>>> 
>>> I have successfully made Ivy read a Eclipse P2 repository in order to download its jars, like any other jar repository, but nothing more. The pieces should be there to do the complete copy chain though.
>> 
>> I'm quite unexperienced with Ivy's Java APIs. Thus, may anybody confirm that this solution is the way to go?
> 
> I think you should look into the "install" feature of Ivy (see the doc about the ant task [1]) and work around that. But as I dig into it, it seems there a missing piece. The loading of an updatesite is coded and probably works, but there is no Ivy resolver which does the actual bridge between Ivy and the update site loader.
> 
> Here is what I would have done:
> 
> 1. Have a ivysettings.xml in which you have both your updatesite and you local ivy repository configured. (here we're missing the update site resolver).

FYI, I've just committed an updatesite resolver:
http://ant.apache.org/ivy/history/trunk/resolver/updatesite.html

I have not tested it yet though.

Nicolas

> 2. In Java, do have an ivy instance:
> Ivy ivy = new Ivy().configure(settingsFile);
> 3. get the updatesite resolver :
> DependencyResolver resolver = ivy.getSettings().getResolver("myUpdatesiteResolverName");
> 4. Loop on every module with resolver.listOrganisations() and resolver.listModules()
> 5. For each module call: ivy.install(...)
> 
> The missing piece should not be hard to code but I don't have much time now. If you're willing to code it, see the OBRResolver for reference.
> 
> Nicolas
> 

Re: Advice for managing large-scale OSGI repository for data analysis

[Nicolas Lalevée <ni...@hibnet.org>]

You main sample didn't through the mailing list, but I wrote some tests as part of Ivy get it working.

> two minor fixes:
> 
> org.apache.ivy.osgi.p2.P2MetadataParser.UnitHandler.handleAttributes(Attributes):
> 
> boolean singleton = Boolean.getBoolean(atts.getValue(SINGLETON)); 
> --> changed to 
> boolean singleton = Boolean.parseBoolean(atts.getValue(SINGLETON));
> 
> Despite that this information is not used, calling Boolean.getBoolean seems to be wrong.

nice catch. fixed.

> in org.apache.ivy.osgi.updatesite.UpdateSiteResolver.init()
> 
> loader.load(u);
> --> changed to:
> RepoDescriptor repoDescriptor = loader.load(u);
> setRepoDescriptor(repoDescriptor);
> 
> otherwise, repoDescriptor gets never initialized.

I did fix that too while making the tests I just wrote pass.

> However, I couldn't make downloading and installing work due to other problems. I tried to follow your advice to iterate over all organisations and from there over all modules. This is the resulting code:
> 
> String resolverName = url.toExternalForm();
> UpdateSiteResolver resolver = new UpdateSiteResolver();
> resolver.setName(resolverName);
> resolver.setUrl(url.toExternalForm());
> for (final OrganisationEntry organisation : resolver.listOrganisations()) {
>   for (final ModuleEntry entry : resolver.listModules(organisation)) {
>     System.out.println(entry);
>   }
> }
> 
> resolver.listOganizations returns an empty list (which is not that amazing since there is no organization attribute).

I fixed that too.
It will now always return an organization with an name as an empty String.

> thus, I tried to use the RepoDescriptor directly :
> 
> RepoDescriptor repo = opt.get();
> for (final ModuleDescriptor module : (Set<ModuleDescriptor>)
> repo.getModules()) {
> ModuleRevisionId moduleRevisionId =
> module.getModuleRevisionId();
> ivy.install(moduleRevisionId, resolverName, "local-disk", new InstallOptions()
> setTransitive(true).setValidate(true).setOverwrite(true));
> }
> }
> 
> This approach fails because Ivy cannot download the Artifacts because there is no URL associated with the artifact.
> Any ideas?

yeah, there was a lot of mess about figuring out the location of the artifacts. The current trunk will behave more nicely.

Nicolas

Re: Advice for managing large-scale OSGI repository for data analysis

[Marcel Bruch <br...@st.informatik.tu-darmstadt.de>]

I attached a minimized main which should make it easier to reproduce the problems.

Re: Advice for managing large-scale OSGI repository for data analysis

[Marcel Bruch <br...@st.informatik.tu-darmstadt.de>]

Hi Nicolas,


two minor fixes:

org.apache.ivy.osgi.p2.P2MetadataParser.UnitHandler.handleAttributes(Attributes):

boolean singleton = Boolean.getBoolean(atts.getValue(SINGLETON)); 
--> changed to 
boolean singleton = Boolean.parseBoolean(atts.getValue(SINGLETON));

Despite that this information is not used, calling Boolean.getBoolean seems to be wrong.


in org.apache.ivy.osgi.updatesite.UpdateSiteResolver.init()

loader.load(u);
 --> changed to:
RepoDescriptor repoDescriptor = loader.load(u);
setRepoDescriptor(repoDescriptor);

otherwise, repoDescriptor gets never initialized.


However, I couldn't make downloading and installing work due to other problems. I tried to follow your advice to iterate over all organisations and from there over all modules. This is the resulting code:

 String resolverName = url.toExternalForm();
 UpdateSiteResolver resolver = new UpdateSiteResolver();
 resolver.setName(resolverName);
 resolver.setUrl(url.toExternalForm());
 for (final OrganisationEntry organisation : resolver.listOrganisations()) {
   for (final ModuleEntry entry : resolver.listModules(organisation)) {
     System.out.println(entry);
   }
 }

resolver.listOganizations returns an empty list (which is not that amazing since there is no organization attribute).


thus, I tried to use the RepoDescriptor directly :

RepoDescriptor repo = opt.get();
for (final ModuleDescriptor module : (Set<ModuleDescriptor>)
 repo.getModules()) {
 ModuleRevisionId moduleRevisionId =
 module.getModuleRevisionId();
 ivy.install(moduleRevisionId, resolverName, "local-disk", new InstallOptions()
 setTransitive(true).setValidate(true).setOverwrite(true));
 }
}

This approach fails because Ivy cannot download the Artifacts because there is no URL associated with the artifact.
Any ideas?


Thanks,
Marcel

Re: Advice for managing large-scale OSGI repository for data analysis

[Marcel Bruch <br...@st.informatik.tu-darmstadt.de>]

Hi Nicolas,

>> 1. Have a ivysettings.xml in which you have both your updatesite and you local ivy repository configured. (here we're missing the update site resolver).
> 
> FYI, I've just committed an updatesite resolver:
> http://ant.apache.org/ivy/history/trunk/resolver/updatesite.html
> 
> I have not tested it yet though.

I'll work on the code repository this weekend. I'll get back to you when I get into trouble when downloading the whole Eclipse Marketplace :)