"Dynamic" org.apache.karaf.features.internal.service.Overrides#override()

[Grzegorz Grzybek <gr...@gmail.com>]

Hello

I'd like to discuss an idea. Currently *only*
org.apache.karaf.features.internal.service.Overrides#shouldOverride()
method decides whether (during feature installation/provisioning) given
bundle/resource URI from a feature should be "overriden" by some slightly
different URI.

By default we can use etc/overrides.properties file to instruct Karaf to
install e.g., mvn:commons-beanutils/commons-beanutils/1.9.3 instead of
1.9.2, but can't tell it to use 1.9.0 instead of 1.8.3. Which is of course
reasonable.

But the easy path scenario is usually not what happens in reality. For
example we may end up with these guava versions installed (real case):
 – mvn:com.google.guava/guava/15.0
 – mvn:com.google.guava/guava/16.0
 – mvn:com.google.guava/guava/16.0.1
 – mvn:com.google.guava/guava/17.0
 – mvn:com.google.guava/guava/18.0
 – mvn:com.google.guava/guava/19.0
 – mvn:com.google.guava/guava/20.0

And from what I've seen, «dependency="true"» is used correctly in majority
of cases, but even if two features from different projects combined in
single distro are used, the order of feature installation matters - we can
end up with either of the bundle with dependency="true" installed (e.g.,
commons-beanutils 1.9.2 or 1.9.3) - but still this can be handled by
etc/overrides.properties.

and what if one feature uses
"mvn:org.apache.geronimo.specs/geronimo-jms_2.0_spec/1.0-alpha-2"
(camel-sjms2) and another one "mvn:javax.jms/javax.jms-api/2.0"
(karaf/jms)? These have different Symbolic-Names... This is problem with
many JavaEE APIs, where we have Javax bundles (where Oracle published a
bundle for API), SMX specs, Geronimo specs, JBoss specs etc.

So my idea is to generalize "overrides" mechanism - change it into kind of
features service hook and provide dedicated, explicit other mechanisms.

For example I could create a maven plugin, that could use some input
information about groupId:artifactId alternatives, some preconfigured
overrides, etc. This maven plugin could be used at karaf-maven-plugin run
time to generate additional ("compiled") file acting as enhanced
etc/overrides.properties.
And this file could be read/used by features service "hook", which could
not only (as Overrides.shouldOverride() does) "translate" bundle URIs, but
change entire features during installation:
 – add some new bundles (configs?) to a feature
 – use different groupId:artifactId of <bundle>
 – skip some bundle
 – ...

Currently Karaf features are NOT like Maven dependencies - Maven builds
transitive graph of dependencies and may choose between deps with same
groupId:artifactId and different version.

With my "hook" idea, static feature definition from *-features.xml file
would only be a "declaration" of a feature, which MAY be changed at runtime.

I imagine that it's not that trivial and features service has huge runtime
part (resolver) that would be (would it?) affected, but I'd like to know
what do you think?

best regards
Grzegorz Grzybek

Re: "Dynamic" org.apache.karaf.features.internal.service.Overrides#override()

[Guillaume Nodet <gn...@apache.org>]

2017-09-14 9:59 GMT+02:00 Grzegorz Grzybek <gr...@gmail.com>:

> Hello
>
> I'd like to discuss an idea. Currently *only*
> org.apache.karaf.features.internal.service.Overrides#shouldOverride()
> method decides whether (during feature installation/provisioning) given
> bundle/resource URI from a feature should be "overriden" by some slightly
> different URI.
>
> By default we can use etc/overrides.properties file to instruct Karaf to
> install e.g., mvn:commons-beanutils/commons-beanutils/1.9.3 instead of
> 1.9.2, but can't tell it to use 1.9.0 instead of 1.8.3. Which is of course
> reasonable.
>

Actually, this is only a default behavior. You can force this case using
 mvn:commons-beanutils/commons-beanutils/1.9.3;range=[1.0,1.9.3)
which would replace any commons-beanutils in the range.
But I think that's slightly irrelevant with the rest of your email.


> But the easy path scenario is usually not what happens in reality. For
> example we may end up with these guava versions installed (real case):
>  – mvn:com.google.guava/guava/15.0
>  – mvn:com.google.guava/guava/16.0
>  – mvn:com.google.guava/guava/16.0.1
>  – mvn:com.google.guava/guava/17.0
>  – mvn:com.google.guava/guava/18.0
>  – mvn:com.google.guava/guava/19.0
>  – mvn:com.google.guava/guava/20.0
>
> And from what I've seen, «dependency="true"» is used correctly in majority
> of cases, but even if two features from different projects combined in
> single distro are used, the order of feature installation matters - we can
> end up with either of the bundle with dependency="true" installed (e.g.,
> commons-beanutils 1.9.2 or 1.9.3) - but still this can be handled by
> etc/overrides.properties.
>
> and what if one feature uses
> "mvn:org.apache.geronimo.specs/geronimo-jms_2.0_spec/1.0-alpha-2"
> (camel-sjms2) and another one "mvn:javax.jms/javax.jms-api/2.0"
> (karaf/jms)? These have different Symbolic-Names... This is problem with
> many JavaEE APIs, where we have Javax bundles (where Oracle published a
> bundle for API), SMX specs, Geronimo specs, JBoss specs etc.
>

Yeah, ideally, those should have a dependency flag to allow them to be
switched, but that may not be always the case.


>
> So my idea is to generalize "overrides" mechanism - change it into kind of
> features service hook and provide dedicated, explicit other mechanisms.
>
> For example I could create a maven plugin, that could use some input
> information about groupId:artifactId alternatives, some preconfigured
> overrides, etc. This maven plugin could be used at karaf-maven-plugin run
> time to generate additional ("compiled") file acting as enhanced
> etc/overrides.properties.
> And this file could be read/used by features service "hook", which could
> not only (as Overrides.shouldOverride() does) "translate" bundle URIs, but
> change entire features during installation:
>  – add some new bundles (configs?) to a feature
>  – use different groupId:artifactId of <bundle>
>  – skip some bundle
>  – ...
>
> Currently Karaf features are NOT like Maven dependencies - Maven builds
> transitive graph of dependencies and may choose between deps with same
> groupId:artifactId and different version.
>
> With my "hook" idea, static feature definition from *-features.xml file
> would only be a "declaration" of a feature, which MAY be changed at
> runtime.
>
> I imagine that it's not that trivial and features service has huge runtime
> part (resolver) that would be (would it?) affected, but I'd like to know
> what do you think?
>

I'm not entirely sure what you have in mind, but what could be quite simple
to implement, without requiring any real resolver change, would be a
service that pre-processes a set of features just before it's used by the
resolver.
The features set is an input to the resolver, it's currently stored at the
following location:

https://github.com/apache/karaf/blob/master/features/core/src/main/java/org/apache/karaf/features/internal/service/Deployer.java#L168

In a usual scenario, the values comes from the following:

https://github.com/apache/karaf/blob/master/features/core/src/main/java/org/apache/karaf/features/internal/service/FeaturesServiceImpl.java#L596-L599

What can be easily achieved, is to pre-process this set through a service
with a simple interface such as
  interface FeaturesProcessor {
    Map<String, Feature> process(Map<String, Feature> features);
  }

Note that we also have a different mechanism that has an effect on features
at runtime, it's the blacklisting policy.
The difference is that this one is actually done when the repositories are
loaded:

https://github.com/apache/karaf/blob/master/features/core/src/main/java/org/apache/karaf/features/internal/service/RepositoryImpl.java#L85

The difference is also because, given the overrides mechanism uses the
resource symbolic name + version to see if an override should take place,
it has to actually load the resources, and *then* eventually override it.

Anyway, big +1 to investigate something around that, as I think it is
clearly missing.
This may also indicates that the fact that application development and
deployment are somewhat conflated in the same xml feature repository need
to be revisited somehow.


> best regards
> Grzegorz Grzybek
>



-- 
------------------------
Guillaume Nodet