OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Raul Kripalani <ra...@apache.org>]

Hi all,

Just to inform you that I've pushed the POM changes to generate the OSGi
Manifest for all Java modules except: gce, cloud, hadoop, log4j2, spark,
yarn, mesos, as the prove a bit more difficult for different reasons each.

Some modules are bundles (e.g. mqtt, zookeeper, kafka, etc.) while others
are fragments (indexing, jta, geospatial, spring, ssh). The criteria to
choose between bundle or fragment was whether or not the module contained
internal processors that ignite-code would have to discover.

I also created a features repository under modules/osgi-karaf/features.

If you perform a complete build of the ignite-1527 branch, you should be
able to install the different modules via the features repository. It has
been tested against Karaf 4.0.2:

    karaf@root()> feature:repo-add
mvn:org.apache.ignite/ignite-osgi-karaf-features/1.5.0-SNAPSHOT/xml/features
    karaf@root()> feature:install ignite-core
    karaf@root()> feature:install ignite-mqtt

The ignite-log4j feature spits out a strange message which I'll debug
tomorrow.

Regards,

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

Re: OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Dmitriy Setrakyan <ds...@apache.org>]

Raul,

As far as I know, here is what you get with bundles:

1. Bundle can be loaded independently from other bundles.

This is false in Ignite. Every optional dependency in Ignite has to be
loaded together with ignite-core module. For example, “ignite-kafka" module
should be a fragment, because it cannot exist without ignite-core module.
The Kafka itself of course should be a separate bundle.


2. Bundles can be loaded with their own class-loaders.

Again, for Ignite optional modules this should not be the case. The
“ignite-kafka” module should be loaded with the same class-loader as
“ignite-core" module.

When you talk about Kafka, and Zookeeper, these dependencies themselves can
be brought as bundles, of course, but “ignite-kafka” and "ignite-zookeeper"
modules should be fragments of the ignite-core bundle.

Are you suggesting the same design? If not, then please explain how having
“ignite-kafka" as a bundle can benefit ignite architecture and ignite users.

D.

On Wed, Nov 11, 2015 at 3:33 PM, Raul Kripalani <ra...@evosent.com> wrote:

> No, Dmitriy. The policy should be to use the right technique for each job,
> and to be respectful with the concepts in the underlying standards.
>
> Fragments and Bundles must be used where they belong and according to their
> roles in the OSGi specification.
>
> Users don't need to know the rationale. They just provision the modules
> onto their containers and that's all.
>
> Fragments are used in very special and sporadic cases, never as a rule.
> Typically to alter some aspect of the host bundle, or to contribute
> internal resources.
>
> That's valid for some modules, but not for others.
>
> There is no point injecting Kafka, MQTT, Hibernate, Zookeeper, etc. into
> Ignite Core through fragments.
>
> You guys ought to stop thinking about classloaders in general, and start
> thinking about building the right modularity with the appropriate import
> and export packages. Classloaders are an internal concept which normally
> doesn't need to permeate to the bundle developer.
> On 11 Nov 2015 21:32, "Dmitriy Setrakyan" <ds...@apache.org> wrote:
>
> > Raul,
> >
> > Personally, I find this policy to be a bit too scientific and it is
> likely
> > to confuse most of our users. From a consistency standpoint why not pick
> > one path, Bundle or Fragment, and roll with it.
> >
> > According to OSGI Wiki [1], a Fragment is something that belongs to a
> > parent bundle and shares the same class-loader as the parent bundle. In
> our
> > case, this is true for all the Ignite dependencies. Wouldn’t it be more
> > appropriate to treat all Ignite dependencies as “Fragments”.
> >
> > [1] http://wiki.osgi.org/wiki/Fragment
> >
> > On Wed, Nov 11, 2015 at 3:41 AM, Raul Kripalani <ra...@apache.org>
> wrote:
> >
> > > Hey Dmitriy,
> > >
> > > On Wed, Nov 11, 2015 at 1:52 AM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > wrote:
> > >
> > > > Can you please describe again the philosophy between choosing a
> > fragment
> > > or
> > > > a bundle for Ignite dependencies. I couldn’t fully grasp it from your
> > > > explanation.
> > > >
> > >
> > > Yes, sorry, I was somewhat brief (it was late here ;-)). The policy
> I've
> > > followed is to create bundles as a first preference. I've resorted to
> > > fragments in the following circumstances:
> > > * Split package situation, e.g. package
> > > org.apache.ignite.internal.processors.query.h2.opt, exported by several
> > > modules with different contents.
> > > * Dynamic discovery, e.g. the one that takes place in
> > > IgniteH2Indexing#createH2SpatialIndex. Since we cannot resolve this
> > > situation using a package import (because there would be multiple
> bundles
> > > exporting that package), therefore by using a fragment we are injecting
> > > these classes into the class-space in the host bundle.
> > > * Modules offering the components listed in IgniteComponentType, which
> > are
> > > discovered dynamically.
> > >
> > > Modules that are represented as fragments are: geospatial, indexing,
> jta,
> > > spring, ssh.
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> >
>

Re: OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Raul Kripalani <ra...@evosent.com>]

No, Dmitriy. The policy should be to use the right technique for each job,
and to be respectful with the concepts in the underlying standards.

Fragments and Bundles must be used where they belong and according to their
roles in the OSGi specification.

Users don't need to know the rationale. They just provision the modules
onto their containers and that's all.

Fragments are used in very special and sporadic cases, never as a rule.
Typically to alter some aspect of the host bundle, or to contribute
internal resources.

That's valid for some modules, but not for others.

There is no point injecting Kafka, MQTT, Hibernate, Zookeeper, etc. into
Ignite Core through fragments.

You guys ought to stop thinking about classloaders in general, and start
thinking about building the right modularity with the appropriate import
and export packages. Classloaders are an internal concept which normally
doesn't need to permeate to the bundle developer.
On 11 Nov 2015 21:32, "Dmitriy Setrakyan" <ds...@apache.org> wrote:

> Raul,
>
> Personally, I find this policy to be a bit too scientific and it is likely
> to confuse most of our users. From a consistency standpoint why not pick
> one path, Bundle or Fragment, and roll with it.
>
> According to OSGI Wiki [1], a Fragment is something that belongs to a
> parent bundle and shares the same class-loader as the parent bundle. In our
> case, this is true for all the Ignite dependencies. Wouldn’t it be more
> appropriate to treat all Ignite dependencies as “Fragments”.
>
> [1] http://wiki.osgi.org/wiki/Fragment
>
> On Wed, Nov 11, 2015 at 3:41 AM, Raul Kripalani <ra...@apache.org> wrote:
>
> > Hey Dmitriy,
> >
> > On Wed, Nov 11, 2015 at 1:52 AM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Can you please describe again the philosophy between choosing a
> fragment
> > or
> > > a bundle for Ignite dependencies. I couldn’t fully grasp it from your
> > > explanation.
> > >
> >
> > Yes, sorry, I was somewhat brief (it was late here ;-)). The policy I've
> > followed is to create bundles as a first preference. I've resorted to
> > fragments in the following circumstances:
> > * Split package situation, e.g. package
> > org.apache.ignite.internal.processors.query.h2.opt, exported by several
> > modules with different contents.
> > * Dynamic discovery, e.g. the one that takes place in
> > IgniteH2Indexing#createH2SpatialIndex. Since we cannot resolve this
> > situation using a package import (because there would be multiple bundles
> > exporting that package), therefore by using a fragment we are injecting
> > these classes into the class-space in the host bundle.
> > * Modules offering the components listed in IgniteComponentType, which
> are
> > discovered dynamically.
> >
> > Modules that are represented as fragments are: geospatial, indexing, jta,
> > spring, ssh.
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
>

Re: OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Dmitriy Setrakyan <ds...@apache.org>]

Raul,

Personally, I find this policy to be a bit too scientific and it is likely
to confuse most of our users. From a consistency standpoint why not pick
one path, Bundle or Fragment, and roll with it.

According to OSGI Wiki [1], a Fragment is something that belongs to a
parent bundle and shares the same class-loader as the parent bundle. In our
case, this is true for all the Ignite dependencies. Wouldn’t it be more
appropriate to treat all Ignite dependencies as “Fragments”.

[1] http://wiki.osgi.org/wiki/Fragment

On Wed, Nov 11, 2015 at 3:41 AM, Raul Kripalani <ra...@apache.org> wrote:

> Hey Dmitriy,
>
> On Wed, Nov 11, 2015 at 1:52 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Can you please describe again the philosophy between choosing a fragment
> or
> > a bundle for Ignite dependencies. I couldn’t fully grasp it from your
> > explanation.
> >
>
> Yes, sorry, I was somewhat brief (it was late here ;-)). The policy I've
> followed is to create bundles as a first preference. I've resorted to
> fragments in the following circumstances:
> * Split package situation, e.g. package
> org.apache.ignite.internal.processors.query.h2.opt, exported by several
> modules with different contents.
> * Dynamic discovery, e.g. the one that takes place in
> IgniteH2Indexing#createH2SpatialIndex. Since we cannot resolve this
> situation using a package import (because there would be multiple bundles
> exporting that package), therefore by using a fragment we are injecting
> these classes into the class-space in the host bundle.
> * Modules offering the components listed in IgniteComponentType, which are
> discovered dynamically.
>
> Modules that are represented as fragments are: geospatial, indexing, jta,
> spring, ssh.
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>

Re: OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Raul Kripalani <ra...@apache.org>]

Hey Dmitriy,

On Wed, Nov 11, 2015 at 1:52 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Can you please describe again the philosophy between choosing a fragment or
> a bundle for Ignite dependencies. I couldn’t fully grasp it from your
> explanation.
>

Yes, sorry, I was somewhat brief (it was late here ;-)). The policy I've
followed is to create bundles as a first preference. I've resorted to
fragments in the following circumstances:
* Split package situation, e.g. package
org.apache.ignite.internal.processors.query.h2.opt, exported by several
modules with different contents.
* Dynamic discovery, e.g. the one that takes place in
IgniteH2Indexing#createH2SpatialIndex. Since we cannot resolve this
situation using a package import (because there would be multiple bundles
exporting that package), therefore by using a fragment we are injecting
these classes into the class-space in the host bundle.
* Modules offering the components listed in IgniteComponentType, which are
discovered dynamically.

Modules that are represented as fragments are: geospatial, indexing, jta,
spring, ssh.

*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

Re: OSGi Manifest headers and feature repository pushed (IGNITE-1527)

[Dmitriy Setrakyan <ds...@apache.org>]

Thanks Raul, this is great!

Can you please describe again the philosophy between choosing a fragment or
a bundle for Ignite dependencies. I couldn’t fully grasp it from your
explanation.

D.

On Tue, Nov 10, 2015 at 4:17 PM, Raul Kripalani <ra...@apache.org> wrote:

> Hi all,
>
> Just to inform you that I've pushed the POM changes to generate the OSGi
> Manifest for all Java modules except: gce, cloud, hadoop, log4j2, spark,
> yarn, mesos, as the prove a bit more difficult for different reasons each.
>
> Some modules are bundles (e.g. mqtt, zookeeper, kafka, etc.) while others
> are fragments (indexing, jta, geospatial, spring, ssh). The criteria to
> choose between bundle or fragment was whether or not the module contained
> internal processors that ignite-code would have to discover.
>
> I also created a features repository under modules/osgi-karaf/features.
>
> If you perform a complete build of the ignite-1527 branch, you should be
> able to install the different modules via the features repository. It has
> been tested against Karaf 4.0.2:
>
>     karaf@root()> feature:repo-add
>
> mvn:org.apache.ignite/ignite-osgi-karaf-features/1.5.0-SNAPSHOT/xml/features
>     karaf@root()> feature:install ignite-core
>     karaf@root()> feature:install ignite-mqtt
>
> The ignite-log4j feature spits out a strange message which I'll debug
> tomorrow.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>