You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Aled Sage <al...@gmail.com> on 2018/09/20 13:53:07 UTC
Rebind historic persisted state: referencing wrap:mvn bundles
Hi all,
TL;DR: I've hit a problem with rebinding to historic persisted state,
when wrap:mvn has been used for OSGi bundles - the symbolic name
changed, so classloading didn't work on rebind. Which of the solutions
should we go for?
_*PROBLEM*_
The persisted state refers to a class in aws-java-sdk 1.11.245, but in
my newer brooklyn I've upgraded this bundle to 1.11.411 (the old bundle
is not there). Because this jar was added using wrap-mvn, the two
versions of the bundle have different symbolic names! Brooklyn therefore
doesn't know to look in the newer aws-java-sdk bundle.
The bundle was added via a feature.xml containing:
<bundle>wrap:mvn:com.amazonaws/aws-java-sdk-bundle/${aws.java.sdk.version}</bundle>
When built locally, this produced a bundle with the very strange
symbolic name:
wrap_file__Users_aledsage_amp_cloudsoft-amp-karaf-5.2.0_system_com_amazonaws_aws-java-sdk-bundle_1.11.245_aws-java-sdk-bundle-1.11.245.jar
_*EXISTING FUNCTIONALITY*_
Brooklyn currently supports a couple of related features:
1. The persisted state will by default not include bundle versions. We
are therefore willing to use newer versions of a given bundle.
2. When adding your own bundle, you can include metadata in it's
MANIFST.mf to say what bundle/version it replaces.
See brooklyn-docs guide/ops/upgrades/_blueprints.md
However, I don't want to use (2) because that would involve fiddling
with the wrap:mvn to add more metadata.
_*POSSIBLE SOLUTIONS*_
There are no doubt many ways to solve this problem. I describe a few of
them below.
I think I favour the class-renames approach because of its simplicity.
*_Add support to class-renames_*
Our deserializingClassRenames.properties allows rebind to handle class
renames, or a specific class moving from one bundle to another. This is
useful for historic persisted state.
We could add support for bundle wildcards, to say that all classes in
one bundle can be loaded from another bundle.
For example:
wrap_blah_aws-java-sdk-bundle-1.11.245.jar\:* :
wrap_blah_aws-java-sdk-bundle-1.11.411.jar\:*
*_Support a bundle-upgrade configuration file_*
We could add support for a config file that gives explicit named bundle
replacements. This would augment the existing functionality (2 above),
but instead of the replacement bundle being defined in the new bundle's
metadata, it could also be defined in this configuration file.
For example, $BROOKLYN_HOME/etc/org.apache.brooklyn.bundleupgrade.cfg
could contain something like:
wrap_blah_aws-java-sdk-bundle-1.11.411.jar:
Brooklyn-Catalog-Upgrade-For-Bundles:
"wrap_blah_aws-java-sdk-bundle-1.11.245.jar: *"
(would need to figure out the cfg versus yaml format here, obviously!)
*_Recognise 'wrap' bundles, and allow newer versions_*
When loading the class, we could recognise the "wrap_" prefix. We could
figure out from the symbolic name what it was built from, and be willing
to use bundles that are newer versions.
However, playing with wrap:mvn, the bundle naming is not as obvious as
one would expect. The symbolic name below is a very different structure
from that above:
wrap_mvn_com.example.brooklyn.test.resources.osgi_brooklyn-test-osgi-com-example-plainoldjar_1.0.0
This example was created in karaf client by running:
bundle:install
wrap:mvn:com.example.brooklyn.test.resources.osgi/brooklyn-test-osgi-com-example-plainoldjar/1.0.0
See brooklyn-server's
org.apache.brooklyn.util.core.ClassLoaderUtils.tryLoadFromBundle.
*_Require users to set the symbolic name, when using wrap:mvn_*
We could require users to *not* use the simple "wrap:mvn", and instead
force them to ensure a more predictable symbolic name + version is used.
However, that sounds harder for users. It also doesn't solve the problem
for anyone with such historic persisted state.
_*LONG TERM / DOCS
*_We should warn people about this in our docs, including a description
of how to work around it.
We should discourage the use of complex types in config keys and
sensors, where we (or the user) don't explicitly control the versioning
and schema of those types.
**_**_
Re: Rebind historic persisted state: referencing wrap:mvn bundles
Posted by Aled Sage <al...@gmail.com>.
Hi all,
I've just been speaking to Alex about this.
Turns out it's more complicated...
The name generated by wrap:mvn will vary depending on the user's runtime
environment. For example, when Alex took a brooklyn build and looked at
the symbolic name of a `wrap:mvn` bundle, it included the runtime path
of his brooklyn install!
---
First, I think we should strongly recommend that people using `wrap:mvn`
specify the symbolic name and version explicitly [1], such as:
wrap:mvn:com.amazonaws/aws-java-sdk-bundle/1.11.411$Bundle-SymbolicName=com.amazonaws.aws-java-sdk-bundle&Bundle-Version=1.11.411
---
If we went down the deserializingClassRenames.properties approach, we'd
need to support wildcards in the bundle name:
wrap_.*_aws-java-sdk-bundle-1.11.245.jar\:* :
com.amazonaws.aws-java-sdk-bundle\:*
That doesn't feel good - evaluating the regex on every class-load will
be inefficient.
---
Another way I'm looking at, and have discussed with Alex, is to include
in the newer version of the bundle the header:
Brooklyn-Catalog-Upgrade-For-Bundles =
wrap:com_amazonaws_aws-java-sdk-bundle:0
(we can add that header to the `wrap:mvn` command, for when we generate
the new version of aws-java-sdk-bundle:1.11.411.)
We already support this header for upgrading explicit bundle names, but
not for auto-generated weird bundle names!
This would tell Brooklyn that our new version of the bundle will upgrade
auto-wrapped bundle of com_amazonaws_aws-java-sdk-bundle.
I'm looking at how easy it would be to add this to
org.apache.brooklyn.core.typereg.BundleUpgradeParser, and whether we'd
avoid the regex evaluation of every class-load.
Aled
[1] https://ops4j1.jira.com/wiki/spaces/paxurl/pages/3833898/Wrap+Protocol
On 10/10/2018 02:24, Paul Campbell wrote:
> Hi all,
>
> I've opened a JIRA ticket for Aled's issue:
> https://issues.apache.org/jira/browse/BROOKLYN-605
>
> I'm also investigation his proposed solution of adding support to
> class-renames here:
> https://github.com/kemitix/brooklyn-server/tree/BROOKLYN-605-rebind-fails-for-historic-state
>
> --
> Paul Campbell
>
>
> On Thu, 20 Sep 2018 at 14:53, Aled Sage <al...@gmail.com> wrote:
>
>> Hi all,
>>
>> TL;DR: I've hit a problem with rebinding to historic persisted state,
>> when wrap:mvn has been used for OSGi bundles - the symbolic name
>> changed, so classloading didn't work on rebind. Which of the solutions
>> should we go for?
>>
>>
>> _*PROBLEM*_
>> The persisted state refers to a class in aws-java-sdk 1.11.245, but in
>> my newer brooklyn I've upgraded this bundle to 1.11.411 (the old bundle
>> is not there). Because this jar was added using wrap-mvn, the two
>> versions of the bundle have different symbolic names! Brooklyn therefore
>> doesn't know to look in the newer aws-java-sdk bundle.
>>
>> The bundle was added via a feature.xml containing:
>>
>>
>> <bundle>wrap:mvn:com.amazonaws/aws-java-sdk-bundle/${aws.java.sdk.version}</bundle>
>>
>> When built locally, this produced a bundle with the very strange
>> symbolic name:
>>
>>
>> wrap_file__Users_aledsage_amp_cloudsoft-amp-karaf-5.2.0_system_com_amazonaws_aws-java-sdk-bundle_1.11.245_aws-java-sdk-bundle-1.11.245.jar
>>
>> _*EXISTING FUNCTIONALITY*_
>> Brooklyn currently supports a couple of related features:
>>
>> 1. The persisted state will by default not include bundle versions. We
>> are therefore willing to use newer versions of a given bundle.
>> 2. When adding your own bundle, you can include metadata in it's
>> MANIFST.mf to say what bundle/version it replaces.
>> See brooklyn-docs guide/ops/upgrades/_blueprints.md
>>
>> However, I don't want to use (2) because that would involve fiddling
>> with the wrap:mvn to add more metadata.
>>
>>
>> _*POSSIBLE SOLUTIONS*_
>> There are no doubt many ways to solve this problem. I describe a few of
>> them below.
>>
>> I think I favour the class-renames approach because of its simplicity.
>>
>> *_Add support to class-renames_*
>> Our deserializingClassRenames.properties allows rebind to handle class
>> renames, or a specific class moving from one bundle to another. This is
>> useful for historic persisted state.
>>
>> We could add support for bundle wildcards, to say that all classes in
>> one bundle can be loaded from another bundle.
>>
>> For example:
>>
>> wrap_blah_aws-java-sdk-bundle-1.11.245.jar\:* :
>> wrap_blah_aws-java-sdk-bundle-1.11.411.jar\:*
>>
>> *_Support a bundle-upgrade configuration file_*
>> We could add support for a config file that gives explicit named bundle
>> replacements. This would augment the existing functionality (2 above),
>> but instead of the replacement bundle being defined in the new bundle's
>> metadata, it could also be defined in this configuration file.
>>
>> For example, $BROOKLYN_HOME/etc/org.apache.brooklyn.bundleupgrade.cfg
>> could contain something like:
>>
>> wrap_blah_aws-java-sdk-bundle-1.11.411.jar:
>> Brooklyn-Catalog-Upgrade-For-Bundles:
>> "wrap_blah_aws-java-sdk-bundle-1.11.245.jar: *"
>>
>> (would need to figure out the cfg versus yaml format here, obviously!)
>>
>> *_Recognise 'wrap' bundles, and allow newer versions_*
>> When loading the class, we could recognise the "wrap_" prefix. We could
>> figure out from the symbolic name what it was built from, and be willing
>> to use bundles that are newer versions.
>>
>> However, playing with wrap:mvn, the bundle naming is not as obvious as
>> one would expect. The symbolic name below is a very different structure
>> from that above:
>>
>>
>> wrap_mvn_com.example.brooklyn.test.resources.osgi_brooklyn-test-osgi-com-example-plainoldjar_1.0.0
>>
>> This example was created in karaf client by running:
>>
>> bundle:install
>>
>> wrap:mvn:com.example.brooklyn.test.resources.osgi/brooklyn-test-osgi-com-example-plainoldjar/1.0.0
>>
>> See brooklyn-server's
>> org.apache.brooklyn.util.core.ClassLoaderUtils.tryLoadFromBundle.
>>
>> *_Require users to set the symbolic name, when using wrap:mvn_*
>> We could require users to *not* use the simple "wrap:mvn", and instead
>> force them to ensure a more predictable symbolic name + version is used.
>>
>> However, that sounds harder for users. It also doesn't solve the problem
>> for anyone with such historic persisted state.
>>
>>
>> _*LONG TERM / DOCS
>> *_We should warn people about this in our docs, including a description
>> of how to work around it.
>>
>> We should discourage the use of complex types in config keys and
>> sensors, where we (or the user) don't explicitly control the versioning
>> and schema of those types.
>>
>> **_**_
>>
>
Re: Rebind historic persisted state: referencing wrap:mvn bundles
Posted by Paul Campbell <pa...@cloudsoftcorp.com>.
Hi all,
I've opened a JIRA ticket for Aled's issue:
https://issues.apache.org/jira/browse/BROOKLYN-605
I'm also investigation his proposed solution of adding support to
class-renames here:
https://github.com/kemitix/brooklyn-server/tree/BROOKLYN-605-rebind-fails-for-historic-state
--
Paul Campbell
On Thu, 20 Sep 2018 at 14:53, Aled Sage <al...@gmail.com> wrote:
> Hi all,
>
> TL;DR: I've hit a problem with rebinding to historic persisted state,
> when wrap:mvn has been used for OSGi bundles - the symbolic name
> changed, so classloading didn't work on rebind. Which of the solutions
> should we go for?
>
>
> _*PROBLEM*_
> The persisted state refers to a class in aws-java-sdk 1.11.245, but in
> my newer brooklyn I've upgraded this bundle to 1.11.411 (the old bundle
> is not there). Because this jar was added using wrap-mvn, the two
> versions of the bundle have different symbolic names! Brooklyn therefore
> doesn't know to look in the newer aws-java-sdk bundle.
>
> The bundle was added via a feature.xml containing:
>
>
> <bundle>wrap:mvn:com.amazonaws/aws-java-sdk-bundle/${aws.java.sdk.version}</bundle>
>
> When built locally, this produced a bundle with the very strange
> symbolic name:
>
>
> wrap_file__Users_aledsage_amp_cloudsoft-amp-karaf-5.2.0_system_com_amazonaws_aws-java-sdk-bundle_1.11.245_aws-java-sdk-bundle-1.11.245.jar
>
> _*EXISTING FUNCTIONALITY*_
> Brooklyn currently supports a couple of related features:
>
> 1. The persisted state will by default not include bundle versions. We
> are therefore willing to use newer versions of a given bundle.
> 2. When adding your own bundle, you can include metadata in it's
> MANIFST.mf to say what bundle/version it replaces.
> See brooklyn-docs guide/ops/upgrades/_blueprints.md
>
> However, I don't want to use (2) because that would involve fiddling
> with the wrap:mvn to add more metadata.
>
>
> _*POSSIBLE SOLUTIONS*_
> There are no doubt many ways to solve this problem. I describe a few of
> them below.
>
> I think I favour the class-renames approach because of its simplicity.
>
> *_Add support to class-renames_*
> Our deserializingClassRenames.properties allows rebind to handle class
> renames, or a specific class moving from one bundle to another. This is
> useful for historic persisted state.
>
> We could add support for bundle wildcards, to say that all classes in
> one bundle can be loaded from another bundle.
>
> For example:
>
> wrap_blah_aws-java-sdk-bundle-1.11.245.jar\:* :
> wrap_blah_aws-java-sdk-bundle-1.11.411.jar\:*
>
> *_Support a bundle-upgrade configuration file_*
> We could add support for a config file that gives explicit named bundle
> replacements. This would augment the existing functionality (2 above),
> but instead of the replacement bundle being defined in the new bundle's
> metadata, it could also be defined in this configuration file.
>
> For example, $BROOKLYN_HOME/etc/org.apache.brooklyn.bundleupgrade.cfg
> could contain something like:
>
> wrap_blah_aws-java-sdk-bundle-1.11.411.jar:
> Brooklyn-Catalog-Upgrade-For-Bundles:
> "wrap_blah_aws-java-sdk-bundle-1.11.245.jar: *"
>
> (would need to figure out the cfg versus yaml format here, obviously!)
>
> *_Recognise 'wrap' bundles, and allow newer versions_*
> When loading the class, we could recognise the "wrap_" prefix. We could
> figure out from the symbolic name what it was built from, and be willing
> to use bundles that are newer versions.
>
> However, playing with wrap:mvn, the bundle naming is not as obvious as
> one would expect. The symbolic name below is a very different structure
> from that above:
>
>
> wrap_mvn_com.example.brooklyn.test.resources.osgi_brooklyn-test-osgi-com-example-plainoldjar_1.0.0
>
> This example was created in karaf client by running:
>
> bundle:install
>
> wrap:mvn:com.example.brooklyn.test.resources.osgi/brooklyn-test-osgi-com-example-plainoldjar/1.0.0
>
> See brooklyn-server's
> org.apache.brooklyn.util.core.ClassLoaderUtils.tryLoadFromBundle.
>
> *_Require users to set the symbolic name, when using wrap:mvn_*
> We could require users to *not* use the simple "wrap:mvn", and instead
> force them to ensure a more predictable symbolic name + version is used.
>
> However, that sounds harder for users. It also doesn't solve the problem
> for anyone with such historic persisted state.
>
>
> _*LONG TERM / DOCS
> *_We should warn people about this in our docs, including a description
> of how to work around it.
>
> We should discourage the use of complex types in config keys and
> sensors, where we (or the user) don't explicitly control the versioning
> and schema of those types.
>
> **_**_
>
--
Paul Campbell
Software Engineer
*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
E: paul.campbell@cloudsoft.io
M: 07476981644 <+447476981644>
T: kemitixcode <https://twitter.com/kemitixcode>
L: https://www.linkedin.com/in/paulkcampbell/
Need a hand with AWS? Get a Free Consultation.
<https://go.cloudsoft.io/healthcheck/>