CamelBlueprintTestSupport doesn't like 6 directories deep

[sohrab <so...@gmail.com>]

I have found a weird/funny bug with CamelBlueprintTestSupport, I thought I
share. I briefly looked at the source code but cannot see any obvious
culprit so I thought I post it here if anyone feels like a challenge.

Basically when I git-clone a project in a directory that is at most 5 levels
deep from root directory, the blueprint test passes. But if it is cloned to
a directory which is 6 levels or more, the blueprint fails to start. (As you
can imagine, it took me ages to figure out this was the case!)

I narrowed down the problem to blueprint config admin not being initialised
properly. Then I looked at the source code for CamelBlueprintTestSupport and
realised it uses hardcoded relative paths like "target/etc".

https://github.com/apache/camel/blob/master/components/camel-test-blueprint/src/main/java/org/apache/camel/test/blueprint/CamelBlueprintTestSupport.java#L523

Here is the thing: I am using Gradle; there is no "target" directory! It
does get created as part of the blueprint test though.

The $1M question is why is 6 directories deep is significant in all this?!
It definitely is not the string length of the absolute path as I tried a
really long path, but only 5 levels deep, and the blueprint started fine.

*If anyone else encountered this, the work-around is to use "target"
directory as the Gradle's buildDir; then the test passes in any directory!*

I'll investigate it further once I get some free time but double you tee
eff!



--
View this message in context: http://camel.465427.n5.nabble.com/CamelBlueprintTestSupport-doesn-t-like-6-directories-deep-tp5787192.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[Brad Johnson <br...@mediadriver.com>]

Ah, that makes sense, I almost mentioned that CBTS uses PojoSR and not a
real OSGi container but thought that was probably irrelevant.  From what
you say here though it does sound relevant.  Just like you can't test with
multiple camel contexts in CBTS.

On Wed, Sep 7, 2016 at 6:45 PM, sohrab <so...@gmail.com> wrote:

> Yeah it is hard one to replicate but once you do, it happens consistently.
>
> I actually got to the bottom of this. So according to the OSGi spec, the
> "entries" inside a bundle are returned in the order they were originally
> attached:
> https://osgi.org/javadoc/r4v43/core/org/osgi/framework/
> Bundle.html#findEntries(java.lang.String,
> java.lang.String, boolean)
>
> Apache Felix Connect respects this. However
> org.ops4j.pax.swissbox.tinybundles.core.metadata.RawBuilder which does the
> attaching, does so in a undeterministic manner by traversing a Set (see my
> last post).
>
> Anyway, this behaviour leads to unreliable tests so I need to find a way
> around it. Fortunately, Apache Aries respects "Bundle-Blueprint" header. So
> if that header is provided, it will not ask Felix to go and search the
> bundle for Blueprints.
>
> *My current, and hopefully definitive, work-around is to customise
> CamelBlueprintHelper.createTestBundle() to add "Bundle-Blueprint" header
> to
> the manifest with a sorted list of blueprints.*
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/CamelBlueprintTestSupport-doesn-t-like-6-directories-
> deep-tp5787192p5787333.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[sohrab <so...@gmail.com>]

Yeah it is hard one to replicate but once you do, it happens consistently.

I actually got to the bottom of this. So according to the OSGi spec, the
"entries" inside a bundle are returned in the order they were originally
attached:
https://osgi.org/javadoc/r4v43/core/org/osgi/framework/Bundle.html#findEntries(java.lang.String,
java.lang.String, boolean)

Apache Felix Connect respects this. However
org.ops4j.pax.swissbox.tinybundles.core.metadata.RawBuilder which does the
attaching, does so in a undeterministic manner by traversing a Set (see my
last post).

Anyway, this behaviour leads to unreliable tests so I need to find a way
around it. Fortunately, Apache Aries respects "Bundle-Blueprint" header. So
if that header is provided, it will not ask Felix to go and search the
bundle for Blueprints.

*My current, and hopefully definitive, work-around is to customise
CamelBlueprintHelper.createTestBundle() to add "Bundle-Blueprint" header to
the manifest with a sorted list of blueprints.*



--
View this message in context: http://camel.465427.n5.nabble.com/CamelBlueprintTestSupport-doesn-t-like-6-directories-deep-tp5787192p5787333.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[Brad Johnson <br...@mediadriver.com>]

That's odd.  I've never run into that problem and I have
blueprint-properties.xml in almost all my bundles. Are your blueprint files
all in the bundles?  Also I have run into problems with the cm-property
place holder name creating problems if the PID has character it doesn't
expect.  I think "-" created a hair pulling problem for me but don't quote
me on that one.  I do know that the cm PID was picky about the name.
Usually I use a dot delimited version of the groupID/artifactID and that
lets me override the properties by putting a groupID/artifactID.cfg file in
the etc directory.

In all my bundles the blueprint files in the OSGI-INF/blueprint folder are
named exactly the same. Since the bundles are loaded into their own
classloaders they don't clash.  The only thing that it does care about is
the PID for the property-placeholder.

Brad

On Wed, Sep 7, 2016 at 2:28 AM, sohrab <so...@gmail.com> wrote:

> Yep, I have all those changes in my version of camel-test-blueprint. But
> anyway I think I narrowed down the problem.
>
> It has nothing to do with Gradle vs. Maven. I have replicated it in both.
> Also the directory path matters but it is not as simple as more or fewer
> than 6.
>
> Basically this is a symptom of using multiple bluerprint XML files where
> one
> defines the cm:property-placeholder which is used by the other blueprint.
>
> Property-placeholders are treated differently during start-up. Unlike bean
> instantiation, that waits till all blueprint XMLs are parsed,
> property-replacement seem to happen right away. This means if a blueprint
> XML that uses a property-placeholder is loaded before the one that defines
> cm:property-placeholder, then the start-up fails. (This is the failure I
> get
> in my tests!)
>
> In Karaf container, the order of Blueprint XML start-up seems to be by the
> alphabetical order of the filenames so it is easy to get around this
> peculiarity.
>
> This does not seem to be true in Apache Felix Connect. If I have to take a
> guess, I'd say it is doing a Map.entrySet() to get blueprints in a loop,
> where the blueprint filename is the map's key and under-the-hood, its hash
> determines the order in the Set.
>
> Why do I think this? Because the created test-bundle uses the absolute path
> of blueprint XML for their names:
>
> blueprint--Users-sohrab-workspaces-client-service-payment-card-v1-target-
> resources-main-OSGI-INF-blueprint-2-camel-context.xml
> blueprint--Users-sohrab-workspaces-client-service-payment-card-v1-target-
> resources-main-OSGI-INF-blueprint-1-config.xml
>
> Where my original blueprint XML names were 1-config.xml and
> 2-camel-context.xml.
>
> This is why the directory path matters. Also why just simply changing
> "build" to "target" which is part of the name seem to change the hash
> enough
> to fix the order, sometimes.
>
> I have further verification of this because RawBuilder does a similar thing
> and actually logs it! So every time the blueprints are out of order in the
> RawBuilder log, I can also reproduce my problem:
>
> http://grepcode.com/file/repo1.maven.org/maven2/org.
> ops4j.pax.swissbox/pax-swissbox-tinybundles/1.3.2/org/ops4j/pax/swissbox/
> tinybundles/core/metadata/RawBuilder.java#77
>
> I hate to change the production code (by combining all blueprints) just to
> cater for a buggy test framework.  I've been looking through the Apache
> Felix Connect and Apache Aries code to figure out where this loading of
> blueprints is happening, to see if I can get around it. If anyone already
> knows, can you please point me to it?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/CamelBlueprintTestSupport-doesn-t-like-6-directories-
> deep-tp5787192p5787264.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[sohrab <so...@gmail.com>]

Yep, I have all those changes in my version of camel-test-blueprint. But
anyway I think I narrowed down the problem.

It has nothing to do with Gradle vs. Maven. I have replicated it in both.
Also the directory path matters but it is not as simple as more or fewer
than 6.

Basically this is a symptom of using multiple bluerprint XML files where one
defines the cm:property-placeholder which is used by the other blueprint. 

Property-placeholders are treated differently during start-up. Unlike bean
instantiation, that waits till all blueprint XMLs are parsed,
property-replacement seem to happen right away. This means if a blueprint
XML that uses a property-placeholder is loaded before the one that defines
cm:property-placeholder, then the start-up fails. (This is the failure I get
in my tests!)

In Karaf container, the order of Blueprint XML start-up seems to be by the
alphabetical order of the filenames so it is easy to get around this
peculiarity. 

This does not seem to be true in Apache Felix Connect. If I have to take a
guess, I'd say it is doing a Map.entrySet() to get blueprints in a loop,
where the blueprint filename is the map's key and under-the-hood, its hash
determines the order in the Set.

Why do I think this? Because the created test-bundle uses the absolute path
of blueprint XML for their names:

blueprint--Users-sohrab-workspaces-client-service-payment-card-v1-target-resources-main-OSGI-INF-blueprint-2-camel-context.xml
blueprint--Users-sohrab-workspaces-client-service-payment-card-v1-target-resources-main-OSGI-INF-blueprint-1-config.xml

Where my original blueprint XML names were 1-config.xml and
2-camel-context.xml.

This is why the directory path matters. Also why just simply changing
"build" to "target" which is part of the name seem to change the hash enough
to fix the order, sometimes.

I have further verification of this because RawBuilder does a similar thing
and actually logs it! So every time the blueprints are out of order in the
RawBuilder log, I can also reproduce my problem:

http://grepcode.com/file/repo1.maven.org/maven2/org.ops4j.pax.swissbox/pax-swissbox-tinybundles/1.3.2/org/ops4j/pax/swissbox/tinybundles/core/metadata/RawBuilder.java#77

I hate to change the production code (by combining all blueprints) just to
cater for a buggy test framework.  I've been looking through the Apache
Felix Connect and Apache Aries code to figure out where this loading of
blueprints is happening, to see if I can get around it. If anyone already
knows, can you please point me to it?



--
View this message in context: http://camel.465427.n5.nabble.com/CamelBlueprintTestSupport-doesn-t-like-6-directories-deep-tp5787192p5787264.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[Brad Johnson <br...@mediadriver.com>]

By the way, which version of CBTS are you using.  There were definitely
flaky issues with it in earlier versions.  Even if you are using earlier
versions of Camel I've found that you can use the later version of CBTS
(like 2.17.3) and it will fix a lot of the problems.

In some of the earliest versions I'd actually have the test loading hang
every now and then.  Obviously an issue with the testing framework at that
point.  But it's really gotten much better in the last few major revs.

On Tue, Sep 6, 2016 at 8:28 PM, sohrab <so...@gmail.com> wrote:

> I should make a correction here. I just found out using "target" directory
> resolves the issues in almost all scenarios (e.g. on our build server and
> most development machines). But even with this fix, I am able reproduce the
> issue on some development machines still.
>
> *On those machines, the work-around is to use a shallower directory.*
>
> So currently I am using
> CamelBlueprintTestSupport.setConfigAdminInitialConfiguration() which is
> part
> of the recent changes made to this class to resolve the race conditions for
> ConfigAdmin initialisation that plagued blueprint testing for years. More
> details here:
> http://ggrzybek.blogspot.com.au/2015/12/camel-blueprint-test-support.html
>
> There is definitely some sort of race condition in that class but it is
> fairly difficult to track down due to asynchronous nature of OSGi service
> loading. (useAsynchronousBlueprintStartup() doesn't help.)
>
>
>
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/CamelBlueprintTestSupport-doesn-t-like-6-directories-
> deep-tp5787192p5787260.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[sohrab <so...@gmail.com>]

I should make a correction here. I just found out using "target" directory
resolves the issues in almost all scenarios (e.g. on our build server and
most development machines). But even with this fix, I am able reproduce the
issue on some development machines still.

*On those machines, the work-around is to use a shallower directory.*

So currently I am using
CamelBlueprintTestSupport.setConfigAdminInitialConfiguration() which is part
of the recent changes made to this class to resolve the race conditions for
ConfigAdmin initialisation that plagued blueprint testing for years. More
details here:
http://ggrzybek.blogspot.com.au/2015/12/camel-blueprint-test-support.html

There is definitely some sort of race condition in that class but it is
fairly difficult to track down due to asynchronous nature of OSGi service
loading. (useAsynchronousBlueprintStartup() doesn't help.)









--
View this message in context: http://camel.465427.n5.nabble.com/CamelBlueprintTestSupport-doesn-t-like-6-directories-deep-tp5787192p5787260.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: CamelBlueprintTestSupport doesn't like 6 directories deep

[Brad Johnson <br...@mediadriver.com>]

Wish I knew Gradle's better but unfortunately don't.


**If anyone else encountered this, the work-around is to use
"target"directory as the Gradle's buildDir; then the test passes in any
directory!**

So no matter what setting the buildDir will work whether you are 5 levels
deep or 15 levels deep?  If you look a the Maven bundle plugin it states
that the build dir defaults to project ${project.build.directory}.   Why
that would change based on what level deep the directory is I haven't a
clue.  But it does make some sense that explicitly specify the buildDir
would resolve issues.  It may be that something in the

Unfortunately my unfamiliarity with Gradle reduces me to hand waving but it
may help.


http://felix.apache.org/components/bundle-plugin/instructions-mojo.html

You can also override the properties configuration in the test itself and
pass them in and that should resolve any issue.  Also if you segregate your
blueprint.xml and create a blueprint-properties.xml in the
src/main/resources they both will load when deployed.  But this permits you
to create a blueprint-properties.xml in the src/test/resources that you can
use at test time instead of the standard properties. In your subclass of
the CamelBlueprintTestSupport you can then specify the following.  In this
case the blueprint-properties.xml is located in your srctest/resources and
will be found by CBTS without specify the exact location.  The same is true
of the OSGI-INF directory which resides in the src/main/resources
directory.  When the test is run both are available for loading (I don't
recall where or if they get moved into the target directory or not.  I
believe they are.)

private static final String TEST_BLUEPRINT_XML =
"blueprint-properties.xml,OSGI-INF/blueprint/blueprint.xml";
@Override protected String getBlueprintDescriptor() { return
TEST_BLUEPRINT_XML; }

On Mon, Sep 5, 2016 at 2:37 AM, sohrab <so...@gmail.com> wrote:

> I have found a weird/funny bug with CamelBlueprintTestSupport, I thought I
> share. I briefly looked at the source code but cannot see any obvious
> culprit so I thought I post it here if anyone feels like a challenge.
>
> Basically when I git-clone a project in a directory that is at most 5
> levels
> deep from root directory, the blueprint test passes. But if it is cloned to
> a directory which is 6 levels or more, the blueprint fails to start. (As
> you
> can imagine, it took me ages to figure out this was the case!)
>
> I narrowed down the problem to blueprint config admin not being initialised
> properly. Then I looked at the source code for CamelBlueprintTestSupport
> and
> realised it uses hardcoded relative paths like "target/etc".
>
> https://github.com/apache/camel/blob/master/components/
> camel-test-blueprint/src/main/java/org/apache/camel/test/blueprint/
> CamelBlueprintTestSupport.java#L523
>
> Here is the thing: I am using Gradle; there is no "target" directory! It
> does get created as part of the blueprint test though.
>
> The $1M question is why is 6 directories deep is significant in all this?!
> It definitely is not the string length of the absolute path as I tried a
> really long path, but only 5 levels deep, and the blueprint started fine.
>
> *If anyone else encountered this, the work-around is to use "target"
> directory as the Gradle's buildDir; then the test passes in any directory!*
>
> I'll investigate it further once I get some free time but double you tee
> eff!
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/CamelBlueprintTestSupport-doesn-t-like-6-directories-
> deep-tp5787192.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>