You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@karaf.apache.org by Guillaume Nodet <gn...@apache.org> on 2014/12/11 09:04:12 UTC
[HEADS UP] Profiles and static distributions
I've just committed to the 4.x branch two new features:
* profiles
* static distributions
Profiles
======
A profile is a data structure that brings together configuration and
provisioning, so it's related to features, but is quite different. A demo
is available at [1].
A profile contains a list of configuration files which can be either
properties based or in any other format. Properties based configurations
are handled in a very specific way, as they can overlay each other (more
below).
One of this properties based configuration holds provisioning informations
along with profiles specific attributes, such as a list of parent profiles,
a list of bundles, a list of features repositories and a list of features.
A profile can have one or more parents, so the hierarchy is an inverted
tree. This allows to define generic profiles and specialise them in
children.
Overlay profile: at some point, there is a need to flatten this hierarchy
into an "overlay" profile. This process is done by walking into the
parents hierarchy (depth first) and computing the resulting configuration
for each configuration file. Non properties configuration files simply
overwrite each other, while properties based configuration complement each
other (with the ability to clear a single value or all values).
Effective profile: properties configurations can contain placeholders to
resolve. The only one defined atm is a property placeholder pointing to a
value in a different configuration. Those placeholders are resolved on the
overlay profile to give the end result effective profile which will be
actually used. An example of using this placeholder is shown in [2] which
end up pointing to the configuration in its parent [3]
Provisioning: each profile contains a list of feature repositories,
features and bundles. Those information will be used in the effective
profile to know which bundles and features are needed on a given instance.
Profiles are not used at runtime for the moment and there are 2 main
usage : creating child instances and generating distributions using the
maven plugin.
Profile manipulation : profiles are manipulated using the Profiles class
(static helpers) and the ProfileBuilder (to build profile instances, which
are immutable). Reading and writing profiles is done using the jdk7 in
FileSystem / Path interface, which obviously provides support for the
standard file system, but could be used with various file systems (inside a
zip [4], readonly github remote repository [5], sftp [6], in memory [7].
Those are the ones I know about). A set of commands can be used to edit
the profiles.
Further considerations: possible improvements include an overlay mechanism
for xml, integration with cellar, file systems using a real git repo,
hazelcast, zookeeper, etc... Applying profiles at runtime could be done
too. Better integration with kars could be provided too.
Static distributions
==============
One use case of profiles is to generate distributions. Changes on the
profile need a rebuild on the distribution (this may or may not be a
problem, depending on your environment, but if it can seems weird in an
OSGi world, it's not so much in a cloud environment). This leads to the
distribution being mostly static, i.e. the user should not manually install
/ uninstall stuff.
A demo is available at [8]
Based on this idea, I improved the maven plugin a bit to allow generating
mostly static distribution : the profiles / features are all installed in
etc/startup.properties. A simplistic read-only configuration admin is used
that will pick up configuration in the etc/ folder directly. This lead to
no runtime dependencies for the provisioning of karaf itself : fileinstall,
the features service can all be removed. The bare framework can only
contain 3 bundles : pax-logging + the static configadmin [9]. One
improvement is that the distribution is generated using reference:file:
urls in startup.properties, avoiding a copy of each jar into the cache. An
additional improvement could be to pre-compute the wiring which can take
some time (but this is not supported by felix).
Thoughts welcomed !
[1] https://github.com/apache/karaf/tree/master/demos/profiles
[2]
https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
[3]
https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/default.profile/version.cfg
[4]
http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html
[5] https://github.com/gnodet/githubfs
[6]
https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09e16f618704
[7] https://github.com/google/jimfs
[8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
[9]
https://github.com/apache/karaf/blob/master/demos/profiles/static/src/main/resources/features.xml
Re: [HEADS UP] Profiles and static distributions
Posted by Achim Nierbeck <bc...@googlemail.com>.
Hi JB,
sounds like a good solution, this way we add some more value to Cave. And
everything is provided from the same technology, so no breach there
especially concerning the git/gist/external repo thing.
+1
regards, Achim
2014-12-15 10:34 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
> Hi guys,
>
> following Guillaume work on the profile, I have an extended proposal that
> I want to discuss with you.
>
> The Karaf profiles have to be stored somewhere: on github/git for
> instance, in order to be applied on a distribution.
> For instance, what I tested is the karaf docker.io feature with profiles:
> it works fine, the docker.io feature bootstraps a docker image with
> Karaf, and I applied a profile on this Karaf instance.
>
> A storage is also needed for the docker.io feature: where the images are
> stored (basically a docker.io github like).
>
> On a private infrastructure, companies don't want to use "external"
> storage (like github): they want to use first their "local" storages. It's
> also valid for hybrid infrastructure: their want to use their local storage
> first (as master), and possible proxy external storages.
>
> Right now, we have Karaf Cave providing:
> - Maven repositories management (stored in Cave or mirroring/proxying
> external repositories)
> - OBR server management
> - Feature Enterprise Repositories management
> Cave is especially interesting with Cellar where all cluster nodes share
> Cave repositories.
>
> In order to have centralised storage (and possibly replicated storage), I
> wonder if we can not extend Cave to provide:
> - Karaf profiles repositories
> - Docker.io images repositories
> It could work like for Maven repository, meaning local storage or
> mirror/proxy to another one.
>
> Like this, we could have Karaf and shell commands to bootstrap Karaf
> docker.io images, or apply profiles on an instance.
>
> WDYT ?
>
> Regards
> JB
>
> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
>
>> I've just committed to the 4.x branch two new features:
>> * profiles
>> * static distributions
>>
>> Profiles
>> ======
>>
>> A profile is a data structure that brings together configuration and
>> provisioning, so it's related to features, but is quite different. A demo
>> is available at [1].
>> A profile contains a list of configuration files which can be either
>> properties based or in any other format. Properties based configurations
>> are handled in a very specific way, as they can overlay each other (more
>> below).
>> One of this properties based configuration holds provisioning informations
>> along with profiles specific attributes, such as a list of parent
>> profiles,
>> a list of bundles, a list of features repositories and a list of features.
>>
>> A profile can have one or more parents, so the hierarchy is an inverted
>> tree. This allows to define generic profiles and specialise them in
>> children.
>>
>> Overlay profile: at some point, there is a need to flatten this hierarchy
>> into an "overlay" profile. This process is done by walking into the
>> parents hierarchy (depth first) and computing the resulting configuration
>> for each configuration file. Non properties configuration files simply
>> overwrite each other, while properties based configuration complement each
>> other (with the ability to clear a single value or all values).
>>
>> Effective profile: properties configurations can contain placeholders to
>> resolve. The only one defined atm is a property placeholder pointing to a
>> value in a different configuration. Those placeholders are resolved on
>> the
>> overlay profile to give the end result effective profile which will be
>> actually used. An example of using this placeholder is shown in [2] which
>> end up pointing to the configuration in its parent [3]
>>
>> Provisioning: each profile contains a list of feature repositories,
>> features and bundles. Those information will be used in the effective
>> profile to know which bundles and features are needed on a given instance.
>> Profiles are not used at runtime for the moment and there are 2 main
>> usage : creating child instances and generating distributions using the
>> maven plugin.
>>
>> Profile manipulation : profiles are manipulated using the Profiles class
>> (static helpers) and the ProfileBuilder (to build profile instances, which
>> are immutable). Reading and writing profiles is done using the jdk7 in
>> FileSystem / Path interface, which obviously provides support for the
>> standard file system, but could be used with various file systems (inside
>> a
>> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
>> Those are the ones I know about). A set of commands can be used to edit
>> the profiles.
>>
>> Further considerations: possible improvements include an overlay mechanism
>> for xml, integration with cellar, file systems using a real git repo,
>> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
>> too. Better integration with kars could be provided too.
>>
>>
>> Static distributions
>> ==============
>>
>> One use case of profiles is to generate distributions. Changes on the
>> profile need a rebuild on the distribution (this may or may not be a
>> problem, depending on your environment, but if it can seems weird in an
>> OSGi world, it's not so much in a cloud environment). This leads to the
>> distribution being mostly static, i.e. the user should not manually
>> install
>> / uninstall stuff.
>> A demo is available at [8]
>> Based on this idea, I improved the maven plugin a bit to allow generating
>> mostly static distribution : the profiles / features are all installed in
>> etc/startup.properties. A simplistic read-only configuration admin is
>> used
>> that will pick up configuration in the etc/ folder directly. This lead to
>> no runtime dependencies for the provisioning of karaf itself :
>> fileinstall,
>> the features service can all be removed. The bare framework can only
>> contain 3 bundles : pax-logging + the static configadmin [9]. One
>> improvement is that the distribution is generated using reference:file:
>> urls in startup.properties, avoiding a copy of each jar into the cache.
>> An
>> additional improvement could be to pre-compute the wiring which can take
>> some time (but this is not supported by felix).
>>
>>
>> Thoughts welcomed !
>>
>>
>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
>> [2]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
>> [3]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/registry/src/main/resources/default.profile/version.cfg
>> [4]
>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
>> fsp/zipfilesystemprovider.html
>> [5] https://github.com/gnodet/githubfs
>> [6]
>> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
>> e16f618704
>> [7] https://github.com/google/jimfs
>> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
>> [9]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/static/src/main/resources/features.xml
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Apache Member
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
Project Lead
blog <http://notizblog.nierbeck.de/>
Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
Software Architect / Project Manager / Scrum Master
Re: [HEADS UP] Profiles and static distributions
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
following Guillaume work on the profile, I have an extended proposal
that I want to discuss with you.
The Karaf profiles have to be stored somewhere: on github/git for
instance, in order to be applied on a distribution.
For instance, what I tested is the karaf docker.io feature with
profiles: it works fine, the docker.io feature bootstraps a docker image
with Karaf, and I applied a profile on this Karaf instance.
A storage is also needed for the docker.io feature: where the images are
stored (basically a docker.io github like).
On a private infrastructure, companies don't want to use "external"
storage (like github): they want to use first their "local" storages.
It's also valid for hybrid infrastructure: their want to use their local
storage first (as master), and possible proxy external storages.
Right now, we have Karaf Cave providing:
- Maven repositories management (stored in Cave or mirroring/proxying
external repositories)
- OBR server management
- Feature Enterprise Repositories management
Cave is especially interesting with Cellar where all cluster nodes share
Cave repositories.
In order to have centralised storage (and possibly replicated storage),
I wonder if we can not extend Cave to provide:
- Karaf profiles repositories
- Docker.io images repositories
It could work like for Maven repository, meaning local storage or
mirror/proxy to another one.
Like this, we could have Karaf and shell commands to bootstrap Karaf
docker.io images, or apply profiles on an instance.
WDYT ?
Regards
JB
On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
> I've just committed to the 4.x branch two new features:
> * profiles
> * static distributions
>
> Profiles
> ======
>
> A profile is a data structure that brings together configuration and
> provisioning, so it's related to features, but is quite different. A demo
> is available at [1].
> A profile contains a list of configuration files which can be either
> properties based or in any other format. Properties based configurations
> are handled in a very specific way, as they can overlay each other (more
> below).
> One of this properties based configuration holds provisioning informations
> along with profiles specific attributes, such as a list of parent profiles,
> a list of bundles, a list of features repositories and a list of features.
>
> A profile can have one or more parents, so the hierarchy is an inverted
> tree. This allows to define generic profiles and specialise them in
> children.
>
> Overlay profile: at some point, there is a need to flatten this hierarchy
> into an "overlay" profile. This process is done by walking into the
> parents hierarchy (depth first) and computing the resulting configuration
> for each configuration file. Non properties configuration files simply
> overwrite each other, while properties based configuration complement each
> other (with the ability to clear a single value or all values).
>
> Effective profile: properties configurations can contain placeholders to
> resolve. The only one defined atm is a property placeholder pointing to a
> value in a different configuration. Those placeholders are resolved on the
> overlay profile to give the end result effective profile which will be
> actually used. An example of using this placeholder is shown in [2] which
> end up pointing to the configuration in its parent [3]
>
> Provisioning: each profile contains a list of feature repositories,
> features and bundles. Those information will be used in the effective
> profile to know which bundles and features are needed on a given instance.
> Profiles are not used at runtime for the moment and there are 2 main
> usage : creating child instances and generating distributions using the
> maven plugin.
>
> Profile manipulation : profiles are manipulated using the Profiles class
> (static helpers) and the ProfileBuilder (to build profile instances, which
> are immutable). Reading and writing profiles is done using the jdk7 in
> FileSystem / Path interface, which obviously provides support for the
> standard file system, but could be used with various file systems (inside a
> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
> Those are the ones I know about). A set of commands can be used to edit
> the profiles.
>
> Further considerations: possible improvements include an overlay mechanism
> for xml, integration with cellar, file systems using a real git repo,
> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
> too. Better integration with kars could be provided too.
>
>
> Static distributions
> ==============
>
> One use case of profiles is to generate distributions. Changes on the
> profile need a rebuild on the distribution (this may or may not be a
> problem, depending on your environment, but if it can seems weird in an
> OSGi world, it's not so much in a cloud environment). This leads to the
> distribution being mostly static, i.e. the user should not manually install
> / uninstall stuff.
> A demo is available at [8]
> Based on this idea, I improved the maven plugin a bit to allow generating
> mostly static distribution : the profiles / features are all installed in
> etc/startup.properties. A simplistic read-only configuration admin is used
> that will pick up configuration in the etc/ folder directly. This lead to
> no runtime dependencies for the provisioning of karaf itself : fileinstall,
> the features service can all be removed. The bare framework can only
> contain 3 bundles : pax-logging + the static configadmin [9]. One
> improvement is that the distribution is generated using reference:file:
> urls in startup.properties, avoiding a copy of each jar into the cache. An
> additional improvement could be to pre-compute the wiring which can take
> some time (but this is not supported by felix).
>
>
> Thoughts welcomed !
>
>
> [1] https://github.com/apache/karaf/tree/master/demos/profiles
> [2]
> https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
> [3]
> https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/default.profile/version.cfg
> [4]
> http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html
> [5] https://github.com/gnodet/githubfs
> [6]
> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09e16f618704
> [7] https://github.com/google/jimfs
> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
> [9]
> https://github.com/apache/karaf/blob/master/demos/profiles/static/src/main/resources/features.xml
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [HEADS UP] Profiles and static distributions
Posted by MrSeltzer <mr...@gmail.com>.
I just stumbled across this, but it may hold some interesting possibilities
or food for thought based on the conversation.
http://mesos.apache.org/documentation/latest/
--
View this message in context: http://karaf.922171.n3.nabble.com/HEADS-UP-Profiles-and-static-distributions-tp4037132p4037337.html
Sent from the Karaf - Dev mailing list archive at Nabble.com.
Re: [HEADS UP] Profiles and static distributions
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Good point Christian for Kubernetes.
I'm preparing/polishing the Docker feature (to leverage what Guillaume
did), I will push on master and send you an e-mail.
Regards
JB
On 12/12/2014 09:09 AM, Christian Schneider wrote:
> I also think the new profiles are a great match for docker images as
> well as other virtualization / cloud environments.
>
> When we start digging into docker we should also take a look at google
> Kubernetes as it might be a good management system for such an environment.
> Redhat as well as many others use it as a basis for their
> deployment/management solutions.
>
> Christian
>
> On 12.12.2014 06:18, Jean-Baptiste Onofré wrote:
>> Hi Guillaume,
>>
>> I took a look this morning and it looks good and I like it.
>>
>> FYI, I started some commands/services to leverage Docker with Karaf, I
>> see a good complement to what you did.
>>
>> Now, we need to update/complete the documentation to explain the users
>> the purposes and how to use it.
>>
>> Thanks !
>> Regards
>> JB
>>
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [HEADS UP] Profiles and static distributions
Posted by Christian Schneider <ch...@die-schneider.net>.
I also think the new profiles are a great match for docker images as
well as other virtualization / cloud environments.
When we start digging into docker we should also take a look at google
Kubernetes as it might be a good management system for such an environment.
Redhat as well as many others use it as a basis for their
deployment/management solutions.
Christian
On 12.12.2014 06:18, Jean-Baptiste Onofré wrote:
> Hi Guillaume,
>
> I took a look this morning and it looks good and I like it.
>
> FYI, I started some commands/services to leverage Docker with Karaf, I
> see a good complement to what you did.
>
> Now, we need to update/complete the documentation to explain the users
> the purposes and how to use it.
>
> Thanks !
> Regards
> JB
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: [HEADS UP] Profiles and static distributions
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Guillaume,
I took a look this morning and it looks good and I like it.
FYI, I started some commands/services to leverage Docker with Karaf, I
see a good complement to what you did.
Now, we need to update/complete the documentation to explain the users
the purposes and how to use it.
Thanks !
Regards
JB
On 12/12/2014 12:46 AM, Guillaume Nodet wrote:
> Those commands already exists.
>
> Yes, the concept is a port of fabric profiles with major simplifications :
> no versioning of profiles, no real runtime registry management and no
> runtime application of profiles to existing containers.
>
> 2014-12-12 0:26 GMT+01:00 Krzysztof Sobkowiak <kr...@gmail.com>:
>
>> Hi Guillaume
>>
>> I can see here the Fabric profiles. Do you plan to implement commands to
>> create/edit/delete profiles too?
>>
>> Regards
>> Krzysztof
>>
>> On 12.12.2014 00:11, Guillaume Nodet wrote:
>>> Profiles are only used to generate a new assembly or child instance.
>>>
>>> The "static" distributions is related to profiles, but can be generated
>> out
>>> of a pure feature list. Such distributions, once started, are usually
>>> limited (my initial goal was to really lock it down), at least that's the
>>> case in the demo i pushed. But this is somewhat controlled by the demo
>>> itself (i.e. the fact that file install + features service aren't
>> installed
>>> is because the plugin is told to use a non standard framework instead of
>>> the default one). So once it has been started, such a static
>> distribution
>>> is "mostly" static, but it's still a valid osgi framework, so if you
>> have a
>>> way to access it (through the locale console, ssh or jmx) you could
>>> manually install the features service bundle, then use it install
>>> additional features. But if everything has been disabled ...
>>>
>>> That said, I suppose profiles could be enhanced to support applying a
>>> profile at runtime, but it opens a whole bunch of different problems.
>>>
>>>
>>> 2014-12-11 22:46 GMT+01:00 Achim Nierbeck <bc...@googlemail.com>:
>>>
>>>> Hi Guillaume,
>>>>
>>>> I'd have to play with it to get the full picture of the consequences.
>>>> But from what I've seen so fare I like the idea.
>>>> Just for clarification, profiles are a "blueprint" for generating a
>> custom
>>>> distribution during build time, but not while running already
>>>> or is it possible to start a "static" karaf container and customize it
>>>> after it has been started?
>>>>
>>>> regards, Achim
>>>>
>>>> 2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
>>>>
>>>>> Thanks Guillaume,
>>>>>
>>>>> I gonna review and update the documentation.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>
>>>>> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
>>>>>
>>>>>> I've just committed to the 4.x branch two new features:
>>>>>> * profiles
>>>>>> * static distributions
>>>>>>
>>>>>> Profiles
>>>>>> ======
>>>>>>
>>>>>> A profile is a data structure that brings together configuration and
>>>>>> provisioning, so it's related to features, but is quite different. A
>>>> demo
>>>>>> is available at [1].
>>>>>> A profile contains a list of configuration files which can be either
>>>>>> properties based or in any other format. Properties based
>>>> configurations
>>>>>> are handled in a very specific way, as they can overlay each other
>> (more
>>>>>> below).
>>>>>> One of this properties based configuration holds provisioning
>>>> informations
>>>>>> along with profiles specific attributes, such as a list of parent
>>>>>> profiles,
>>>>>> a list of bundles, a list of features repositories and a list of
>>>> features.
>>>>>> A profile can have one or more parents, so the hierarchy is an
>> inverted
>>>>>> tree. This allows to define generic profiles and specialise them in
>>>>>> children.
>>>>>>
>>>>>> Overlay profile: at some point, there is a need to flatten this
>>>> hierarchy
>>>>>> into an "overlay" profile. This process is done by walking into the
>>>>>> parents hierarchy (depth first) and computing the resulting
>>>> configuration
>>>>>> for each configuration file. Non properties configuration files
>> simply
>>>>>> overwrite each other, while properties based configuration complement
>>>> each
>>>>>> other (with the ability to clear a single value or all values).
>>>>>>
>>>>>> Effective profile: properties configurations can contain placeholders
>> to
>>>>>> resolve. The only one defined atm is a property placeholder pointing
>> to
>>>> a
>>>>>> value in a different configuration. Those placeholders are resolved
>> on
>>>>>> the
>>>>>> overlay profile to give the end result effective profile which will be
>>>>>> actually used. An example of using this placeholder is shown in [2]
>>>> which
>>>>>> end up pointing to the configuration in its parent [3]
>>>>>>
>>>>>> Provisioning: each profile contains a list of feature repositories,
>>>>>> features and bundles. Those information will be used in the effective
>>>>>> profile to know which bundles and features are needed on a given
>>>> instance.
>>>>>> Profiles are not used at runtime for the moment and there are 2
>> main
>>>>>> usage : creating child instances and generating distributions using
>> the
>>>>>> maven plugin.
>>>>>>
>>>>>> Profile manipulation : profiles are manipulated using the Profiles
>> class
>>>>>> (static helpers) and the ProfileBuilder (to build profile instances,
>>>> which
>>>>>> are immutable). Reading and writing profiles is done using the jdk7
>> in
>>>>>> FileSystem / Path interface, which obviously provides support for the
>>>>>> standard file system, but could be used with various file systems
>>>> (inside
>>>>>> a
>>>>>> zip [4], readonly github remote repository [5], sftp [6], in memory
>> [7].
>>>>>> Those are the ones I know about). A set of commands can be used to
>> edit
>>>>>> the profiles.
>>>>>>
>>>>>> Further considerations: possible improvements include an overlay
>>>> mechanism
>>>>>> for xml, integration with cellar, file systems using a real git repo,
>>>>>> hazelcast, zookeeper, etc... Applying profiles at runtime could be
>> done
>>>>>> too. Better integration with kars could be provided too.
>>>>>>
>>>>>>
>>>>>> Static distributions
>>>>>> ==============
>>>>>>
>>>>>> One use case of profiles is to generate distributions. Changes on the
>>>>>> profile need a rebuild on the distribution (this may or may not be a
>>>>>> problem, depending on your environment, but if it can seems weird in
>> an
>>>>>> OSGi world, it's not so much in a cloud environment). This leads to
>> the
>>>>>> distribution being mostly static, i.e. the user should not manually
>>>>>> install
>>>>>> / uninstall stuff.
>>>>>> A demo is available at [8]
>>>>>> Based on this idea, I improved the maven plugin a bit to allow
>>>> generating
>>>>>> mostly static distribution : the profiles / features are all installed
>>>> in
>>>>>> etc/startup.properties. A simplistic read-only configuration admin is
>>>>>> used
>>>>>> that will pick up configuration in the etc/ folder directly. This
>> lead
>>>> to
>>>>>> no runtime dependencies for the provisioning of karaf itself :
>>>>>> fileinstall,
>>>>>> the features service can all be removed. The bare framework can only
>>>>>> contain 3 bundles : pax-logging + the static configadmin [9]. One
>>>>>> improvement is that the distribution is generated using
>> reference:file:
>>>>>> urls in startup.properties, avoiding a copy of each jar into the
>> cache.
>>>>>> An
>>>>>> additional improvement could be to pre-compute the wiring which can
>> take
>>>>>> some time (but this is not supported by felix).
>>>>>>
>>>>>>
>>>>>> Thoughts welcomed !
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
>>>>>> [2]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
>>>>>> [3]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/registry/src/main/resources/default.profile/version.cfg
>>>>>> [4]
>>>>>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
>>>>>> fsp/zipfilesystemprovider.html
>>>>>> [5] https://github.com/gnodet/githubfs
>>>>>> [6]
>>>>>>
>>>>
>> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
>>>>>> e16f618704
>>>>>> [7] https://github.com/google/jimfs
>>>>>> [8]
>> https://github.com/apache/karaf/blob/master/demos/profiles/static/
>>>>>> [9]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/static/src/main/resources/features.xml
>>>>>>
>>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Apache Member
>>>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>>>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>> Committer &
>>>> Project Lead
>>>> blog <http://notizblog.nierbeck.de/>
>>>> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
>>>>
>>>> Software Architect / Project Manager / Scrum Master
>>>>
>>
>>
>> --
>> Krzysztof Sobkowiak
>>
>> JEE & OSS Architect | Senior Solution Architect @ Capgemini | Committer
>> @ ASF
>> Capgemini <http://www.pl.capgemini.com/> | Software Solutions Center
>> <http://www.pl.capgemini-sdm.com/> | Wroclaw
>> e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
>> Twitter: @KSobkowiak
>> Calendar: http://goo.gl/yvsebC
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [HEADS UP] Profiles and static distributions
Posted by Sobkowiak Krzysztof <kr...@gmail.com>.
Indeed. I haven't seen them before. I think it will be a great feature
together with Docker and Kubernetes support
Regards
Krzysztof
On 12.12.2014 00:46, Guillaume Nodet wrote:
> Those commands already exists.
>
> Yes, the concept is a port of fabric profiles with major simplifications :
> no versioning of profiles, no real runtime registry management and no
> runtime application of profiles to existing containers.
>
> 2014-12-12 0:26 GMT+01:00 Krzysztof Sobkowiak <kr...@gmail.com>:
>
>> Hi Guillaume
>>
>> I can see here the Fabric profiles. Do you plan to implement commands to
>> create/edit/delete profiles too?
>>
>> Regards
>> Krzysztof
>>
>> On 12.12.2014 00:11, Guillaume Nodet wrote:
>>> Profiles are only used to generate a new assembly or child instance.
>>>
>>> The "static" distributions is related to profiles, but can be generated
>> out
>>> of a pure feature list. Such distributions, once started, are usually
>>> limited (my initial goal was to really lock it down), at least that's the
>>> case in the demo i pushed. But this is somewhat controlled by the demo
>>> itself (i.e. the fact that file install + features service aren't
>> installed
>>> is because the plugin is told to use a non standard framework instead of
>>> the default one). So once it has been started, such a static
>> distribution
>>> is "mostly" static, but it's still a valid osgi framework, so if you
>> have a
>>> way to access it (through the locale console, ssh or jmx) you could
>>> manually install the features service bundle, then use it install
>>> additional features. But if everything has been disabled ...
>>>
>>> That said, I suppose profiles could be enhanced to support applying a
>>> profile at runtime, but it opens a whole bunch of different problems.
>>>
>>>
>>> 2014-12-11 22:46 GMT+01:00 Achim Nierbeck <bc...@googlemail.com>:
>>>
>>>> Hi Guillaume,
>>>>
>>>> I'd have to play with it to get the full picture of the consequences.
>>>> But from what I've seen so fare I like the idea.
>>>> Just for clarification, profiles are a "blueprint" for generating a
>> custom
>>>> distribution during build time, but not while running already
>>>> or is it possible to start a "static" karaf container and customize it
>>>> after it has been started?
>>>>
>>>> regards, Achim
>>>>
>>>> 2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
>>>>
>>>>> Thanks Guillaume,
>>>>>
>>>>> I gonna review and update the documentation.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>
>>>>> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
>>>>>
>>>>>> I've just committed to the 4.x branch two new features:
>>>>>> * profiles
>>>>>> * static distributions
>>>>>>
>>>>>> Profiles
>>>>>> ======
>>>>>>
>>>>>> A profile is a data structure that brings together configuration and
>>>>>> provisioning, so it's related to features, but is quite different. A
>>>> demo
>>>>>> is available at [1].
>>>>>> A profile contains a list of configuration files which can be either
>>>>>> properties based or in any other format. Properties based
>>>> configurations
>>>>>> are handled in a very specific way, as they can overlay each other
>> (more
>>>>>> below).
>>>>>> One of this properties based configuration holds provisioning
>>>> informations
>>>>>> along with profiles specific attributes, such as a list of parent
>>>>>> profiles,
>>>>>> a list of bundles, a list of features repositories and a list of
>>>> features.
>>>>>> A profile can have one or more parents, so the hierarchy is an
>> inverted
>>>>>> tree. This allows to define generic profiles and specialise them in
>>>>>> children.
>>>>>>
>>>>>> Overlay profile: at some point, there is a need to flatten this
>>>> hierarchy
>>>>>> into an "overlay" profile. This process is done by walking into the
>>>>>> parents hierarchy (depth first) and computing the resulting
>>>> configuration
>>>>>> for each configuration file. Non properties configuration files
>> simply
>>>>>> overwrite each other, while properties based configuration complement
>>>> each
>>>>>> other (with the ability to clear a single value or all values).
>>>>>>
>>>>>> Effective profile: properties configurations can contain placeholders
>> to
>>>>>> resolve. The only one defined atm is a property placeholder pointing
>> to
>>>> a
>>>>>> value in a different configuration. Those placeholders are resolved
>> on
>>>>>> the
>>>>>> overlay profile to give the end result effective profile which will be
>>>>>> actually used. An example of using this placeholder is shown in [2]
>>>> which
>>>>>> end up pointing to the configuration in its parent [3]
>>>>>>
>>>>>> Provisioning: each profile contains a list of feature repositories,
>>>>>> features and bundles. Those information will be used in the effective
>>>>>> profile to know which bundles and features are needed on a given
>>>> instance.
>>>>>> Profiles are not used at runtime for the moment and there are 2
>> main
>>>>>> usage : creating child instances and generating distributions using
>> the
>>>>>> maven plugin.
>>>>>>
>>>>>> Profile manipulation : profiles are manipulated using the Profiles
>> class
>>>>>> (static helpers) and the ProfileBuilder (to build profile instances,
>>>> which
>>>>>> are immutable). Reading and writing profiles is done using the jdk7
>> in
>>>>>> FileSystem / Path interface, which obviously provides support for the
>>>>>> standard file system, but could be used with various file systems
>>>> (inside
>>>>>> a
>>>>>> zip [4], readonly github remote repository [5], sftp [6], in memory
>> [7].
>>>>>> Those are the ones I know about). A set of commands can be used to
>> edit
>>>>>> the profiles.
>>>>>>
>>>>>> Further considerations: possible improvements include an overlay
>>>> mechanism
>>>>>> for xml, integration with cellar, file systems using a real git repo,
>>>>>> hazelcast, zookeeper, etc... Applying profiles at runtime could be
>> done
>>>>>> too. Better integration with kars could be provided too.
>>>>>>
>>>>>>
>>>>>> Static distributions
>>>>>> ==============
>>>>>>
>>>>>> One use case of profiles is to generate distributions. Changes on the
>>>>>> profile need a rebuild on the distribution (this may or may not be a
>>>>>> problem, depending on your environment, but if it can seems weird in
>> an
>>>>>> OSGi world, it's not so much in a cloud environment). This leads to
>> the
>>>>>> distribution being mostly static, i.e. the user should not manually
>>>>>> install
>>>>>> / uninstall stuff.
>>>>>> A demo is available at [8]
>>>>>> Based on this idea, I improved the maven plugin a bit to allow
>>>> generating
>>>>>> mostly static distribution : the profiles / features are all installed
>>>> in
>>>>>> etc/startup.properties. A simplistic read-only configuration admin is
>>>>>> used
>>>>>> that will pick up configuration in the etc/ folder directly. This
>> lead
>>>> to
>>>>>> no runtime dependencies for the provisioning of karaf itself :
>>>>>> fileinstall,
>>>>>> the features service can all be removed. The bare framework can only
>>>>>> contain 3 bundles : pax-logging + the static configadmin [9]. One
>>>>>> improvement is that the distribution is generated using
>> reference:file:
>>>>>> urls in startup.properties, avoiding a copy of each jar into the
>> cache.
>>>>>> An
>>>>>> additional improvement could be to pre-compute the wiring which can
>> take
>>>>>> some time (but this is not supported by felix).
>>>>>>
>>>>>>
>>>>>> Thoughts welcomed !
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
>>>>>> [2]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
>>>>>> [3]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/registry/src/main/resources/default.profile/version.cfg
>>>>>> [4]
>>>>>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
>>>>>> fsp/zipfilesystemprovider.html
>>>>>> [5] https://github.com/gnodet/githubfs
>>>>>> [6]
>>>>>>
>> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
>>>>>> e16f618704
>>>>>> [7] https://github.com/google/jimfs
>>>>>> [8]
>> https://github.com/apache/karaf/blob/master/demos/profiles/static/
>>>>>> [9]
>>>>>> https://github.com/apache/karaf/blob/master/demos/
>>>>>> profiles/static/src/main/resources/features.xml
>>>>>>
>>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
>>>> --
>>>>
>>>> Apache Member
>>>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>>>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
>> Committer &
>>>> Project Lead
>>>> blog <http://notizblog.nierbeck.de/>
>>>> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
>>>>
>>>> Software Architect / Project Manager / Scrum Master
>>>>
>>
>> --
>> Krzysztof Sobkowiak
>>
>> JEE & OSS Architect | Senior Solution Architect @ Capgemini | Committer
>> @ ASF
>> Capgemini <http://www.pl.capgemini.com/> | Software Solutions Center
>> <http://www.pl.capgemini-sdm.com/> | Wroclaw
>> e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
>> Twitter: @KSobkowiak
>> Calendar: http://goo.gl/yvsebC
>>
--
Krzysztof Sobkowiak
JEE & OSS Architect | Technical Architect @ Capgemini | Committer @ ASF
Capgemini <http://www.pl.capgemini.com/> | Software Solutions Center
<http://www.pl.capgemini-sdm.com/> | Wroclaw
e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
Twitter: @KSobkowiak
Calendar: http://goo.gl/yvsebC
Re: [HEADS UP] Profiles and static distributions
Posted by Guillaume Nodet <gn...@apache.org>.
Those commands already exists.
Yes, the concept is a port of fabric profiles with major simplifications :
no versioning of profiles, no real runtime registry management and no
runtime application of profiles to existing containers.
2014-12-12 0:26 GMT+01:00 Krzysztof Sobkowiak <kr...@gmail.com>:
> Hi Guillaume
>
> I can see here the Fabric profiles. Do you plan to implement commands to
> create/edit/delete profiles too?
>
> Regards
> Krzysztof
>
> On 12.12.2014 00:11, Guillaume Nodet wrote:
> > Profiles are only used to generate a new assembly or child instance.
> >
> > The "static" distributions is related to profiles, but can be generated
> out
> > of a pure feature list. Such distributions, once started, are usually
> > limited (my initial goal was to really lock it down), at least that's the
> > case in the demo i pushed. But this is somewhat controlled by the demo
> > itself (i.e. the fact that file install + features service aren't
> installed
> > is because the plugin is told to use a non standard framework instead of
> > the default one). So once it has been started, such a static
> distribution
> > is "mostly" static, but it's still a valid osgi framework, so if you
> have a
> > way to access it (through the locale console, ssh or jmx) you could
> > manually install the features service bundle, then use it install
> > additional features. But if everything has been disabled ...
> >
> > That said, I suppose profiles could be enhanced to support applying a
> > profile at runtime, but it opens a whole bunch of different problems.
> >
> >
> > 2014-12-11 22:46 GMT+01:00 Achim Nierbeck <bc...@googlemail.com>:
> >
> >> Hi Guillaume,
> >>
> >> I'd have to play with it to get the full picture of the consequences.
> >> But from what I've seen so fare I like the idea.
> >> Just for clarification, profiles are a "blueprint" for generating a
> custom
> >> distribution during build time, but not while running already
> >> or is it possible to start a "static" karaf container and customize it
> >> after it has been started?
> >>
> >> regards, Achim
> >>
> >> 2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
> >>
> >>> Thanks Guillaume,
> >>>
> >>> I gonna review and update the documentation.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>>
> >>> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
> >>>
> >>>> I've just committed to the 4.x branch two new features:
> >>>> * profiles
> >>>> * static distributions
> >>>>
> >>>> Profiles
> >>>> ======
> >>>>
> >>>> A profile is a data structure that brings together configuration and
> >>>> provisioning, so it's related to features, but is quite different. A
> >> demo
> >>>> is available at [1].
> >>>> A profile contains a list of configuration files which can be either
> >>>> properties based or in any other format. Properties based
> >> configurations
> >>>> are handled in a very specific way, as they can overlay each other
> (more
> >>>> below).
> >>>> One of this properties based configuration holds provisioning
> >> informations
> >>>> along with profiles specific attributes, such as a list of parent
> >>>> profiles,
> >>>> a list of bundles, a list of features repositories and a list of
> >> features.
> >>>> A profile can have one or more parents, so the hierarchy is an
> inverted
> >>>> tree. This allows to define generic profiles and specialise them in
> >>>> children.
> >>>>
> >>>> Overlay profile: at some point, there is a need to flatten this
> >> hierarchy
> >>>> into an "overlay" profile. This process is done by walking into the
> >>>> parents hierarchy (depth first) and computing the resulting
> >> configuration
> >>>> for each configuration file. Non properties configuration files
> simply
> >>>> overwrite each other, while properties based configuration complement
> >> each
> >>>> other (with the ability to clear a single value or all values).
> >>>>
> >>>> Effective profile: properties configurations can contain placeholders
> to
> >>>> resolve. The only one defined atm is a property placeholder pointing
> to
> >> a
> >>>> value in a different configuration. Those placeholders are resolved
> on
> >>>> the
> >>>> overlay profile to give the end result effective profile which will be
> >>>> actually used. An example of using this placeholder is shown in [2]
> >> which
> >>>> end up pointing to the configuration in its parent [3]
> >>>>
> >>>> Provisioning: each profile contains a list of feature repositories,
> >>>> features and bundles. Those information will be used in the effective
> >>>> profile to know which bundles and features are needed on a given
> >> instance.
> >>>> Profiles are not used at runtime for the moment and there are 2
> main
> >>>> usage : creating child instances and generating distributions using
> the
> >>>> maven plugin.
> >>>>
> >>>> Profile manipulation : profiles are manipulated using the Profiles
> class
> >>>> (static helpers) and the ProfileBuilder (to build profile instances,
> >> which
> >>>> are immutable). Reading and writing profiles is done using the jdk7
> in
> >>>> FileSystem / Path interface, which obviously provides support for the
> >>>> standard file system, but could be used with various file systems
> >> (inside
> >>>> a
> >>>> zip [4], readonly github remote repository [5], sftp [6], in memory
> [7].
> >>>> Those are the ones I know about). A set of commands can be used to
> edit
> >>>> the profiles.
> >>>>
> >>>> Further considerations: possible improvements include an overlay
> >> mechanism
> >>>> for xml, integration with cellar, file systems using a real git repo,
> >>>> hazelcast, zookeeper, etc... Applying profiles at runtime could be
> done
> >>>> too. Better integration with kars could be provided too.
> >>>>
> >>>>
> >>>> Static distributions
> >>>> ==============
> >>>>
> >>>> One use case of profiles is to generate distributions. Changes on the
> >>>> profile need a rebuild on the distribution (this may or may not be a
> >>>> problem, depending on your environment, but if it can seems weird in
> an
> >>>> OSGi world, it's not so much in a cloud environment). This leads to
> the
> >>>> distribution being mostly static, i.e. the user should not manually
> >>>> install
> >>>> / uninstall stuff.
> >>>> A demo is available at [8]
> >>>> Based on this idea, I improved the maven plugin a bit to allow
> >> generating
> >>>> mostly static distribution : the profiles / features are all installed
> >> in
> >>>> etc/startup.properties. A simplistic read-only configuration admin is
> >>>> used
> >>>> that will pick up configuration in the etc/ folder directly. This
> lead
> >> to
> >>>> no runtime dependencies for the provisioning of karaf itself :
> >>>> fileinstall,
> >>>> the features service can all be removed. The bare framework can only
> >>>> contain 3 bundles : pax-logging + the static configadmin [9]. One
> >>>> improvement is that the distribution is generated using
> reference:file:
> >>>> urls in startup.properties, avoiding a copy of each jar into the
> cache.
> >>>> An
> >>>> additional improvement could be to pre-compute the wiring which can
> take
> >>>> some time (but this is not supported by felix).
> >>>>
> >>>>
> >>>> Thoughts welcomed !
> >>>>
> >>>>
> >>>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
> >>>> [2]
> >>>> https://github.com/apache/karaf/blob/master/demos/
> >>>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
> >>>> [3]
> >>>> https://github.com/apache/karaf/blob/master/demos/
> >>>> profiles/registry/src/main/resources/default.profile/version.cfg
> >>>> [4]
> >>>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
> >>>> fsp/zipfilesystemprovider.html
> >>>> [5] https://github.com/gnodet/githubfs
> >>>> [6]
> >>>>
> >>
> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
> >>>> e16f618704
> >>>> [7] https://github.com/google/jimfs
> >>>> [8]
> https://github.com/apache/karaf/blob/master/demos/profiles/static/
> >>>> [9]
> >>>> https://github.com/apache/karaf/blob/master/demos/
> >>>> profiles/static/src/main/resources/features.xml
> >>>>
> >>>>
> >>> --
> >>> Jean-Baptiste Onofré
> >>> jbonofre@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>>
> >>
> >>
> >> --
> >>
> >> Apache Member
> >> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> >> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/>
> Committer &
> >> Project Lead
> >> blog <http://notizblog.nierbeck.de/>
> >> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
> >>
> >> Software Architect / Project Manager / Scrum Master
> >>
>
>
> --
> Krzysztof Sobkowiak
>
> JEE & OSS Architect | Senior Solution Architect @ Capgemini | Committer
> @ ASF
> Capgemini <http://www.pl.capgemini.com/> | Software Solutions Center
> <http://www.pl.capgemini-sdm.com/> | Wroclaw
> e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
> Twitter: @KSobkowiak
> Calendar: http://goo.gl/yvsebC
>
Re: [HEADS UP] Profiles and static distributions
Posted by Krzysztof Sobkowiak <kr...@gmail.com>.
Hi Guillaume
I can see here the Fabric profiles. Do you plan to implement commands to
create/edit/delete profiles too?
Regards
Krzysztof
On 12.12.2014 00:11, Guillaume Nodet wrote:
> Profiles are only used to generate a new assembly or child instance.
>
> The "static" distributions is related to profiles, but can be generated out
> of a pure feature list. Such distributions, once started, are usually
> limited (my initial goal was to really lock it down), at least that's the
> case in the demo i pushed. But this is somewhat controlled by the demo
> itself (i.e. the fact that file install + features service aren't installed
> is because the plugin is told to use a non standard framework instead of
> the default one). So once it has been started, such a static distribution
> is "mostly" static, but it's still a valid osgi framework, so if you have a
> way to access it (through the locale console, ssh or jmx) you could
> manually install the features service bundle, then use it install
> additional features. But if everything has been disabled ...
>
> That said, I suppose profiles could be enhanced to support applying a
> profile at runtime, but it opens a whole bunch of different problems.
>
>
> 2014-12-11 22:46 GMT+01:00 Achim Nierbeck <bc...@googlemail.com>:
>
>> Hi Guillaume,
>>
>> I'd have to play with it to get the full picture of the consequences.
>> But from what I've seen so fare I like the idea.
>> Just for clarification, profiles are a "blueprint" for generating a custom
>> distribution during build time, but not while running already
>> or is it possible to start a "static" karaf container and customize it
>> after it has been started?
>>
>> regards, Achim
>>
>> 2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
>>
>>> Thanks Guillaume,
>>>
>>> I gonna review and update the documentation.
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
>>>
>>>> I've just committed to the 4.x branch two new features:
>>>> * profiles
>>>> * static distributions
>>>>
>>>> Profiles
>>>> ======
>>>>
>>>> A profile is a data structure that brings together configuration and
>>>> provisioning, so it's related to features, but is quite different. A
>> demo
>>>> is available at [1].
>>>> A profile contains a list of configuration files which can be either
>>>> properties based or in any other format. Properties based
>> configurations
>>>> are handled in a very specific way, as they can overlay each other (more
>>>> below).
>>>> One of this properties based configuration holds provisioning
>> informations
>>>> along with profiles specific attributes, such as a list of parent
>>>> profiles,
>>>> a list of bundles, a list of features repositories and a list of
>> features.
>>>> A profile can have one or more parents, so the hierarchy is an inverted
>>>> tree. This allows to define generic profiles and specialise them in
>>>> children.
>>>>
>>>> Overlay profile: at some point, there is a need to flatten this
>> hierarchy
>>>> into an "overlay" profile. This process is done by walking into the
>>>> parents hierarchy (depth first) and computing the resulting
>> configuration
>>>> for each configuration file. Non properties configuration files simply
>>>> overwrite each other, while properties based configuration complement
>> each
>>>> other (with the ability to clear a single value or all values).
>>>>
>>>> Effective profile: properties configurations can contain placeholders to
>>>> resolve. The only one defined atm is a property placeholder pointing to
>> a
>>>> value in a different configuration. Those placeholders are resolved on
>>>> the
>>>> overlay profile to give the end result effective profile which will be
>>>> actually used. An example of using this placeholder is shown in [2]
>> which
>>>> end up pointing to the configuration in its parent [3]
>>>>
>>>> Provisioning: each profile contains a list of feature repositories,
>>>> features and bundles. Those information will be used in the effective
>>>> profile to know which bundles and features are needed on a given
>> instance.
>>>> Profiles are not used at runtime for the moment and there are 2 main
>>>> usage : creating child instances and generating distributions using the
>>>> maven plugin.
>>>>
>>>> Profile manipulation : profiles are manipulated using the Profiles class
>>>> (static helpers) and the ProfileBuilder (to build profile instances,
>> which
>>>> are immutable). Reading and writing profiles is done using the jdk7 in
>>>> FileSystem / Path interface, which obviously provides support for the
>>>> standard file system, but could be used with various file systems
>> (inside
>>>> a
>>>> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
>>>> Those are the ones I know about). A set of commands can be used to edit
>>>> the profiles.
>>>>
>>>> Further considerations: possible improvements include an overlay
>> mechanism
>>>> for xml, integration with cellar, file systems using a real git repo,
>>>> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
>>>> too. Better integration with kars could be provided too.
>>>>
>>>>
>>>> Static distributions
>>>> ==============
>>>>
>>>> One use case of profiles is to generate distributions. Changes on the
>>>> profile need a rebuild on the distribution (this may or may not be a
>>>> problem, depending on your environment, but if it can seems weird in an
>>>> OSGi world, it's not so much in a cloud environment). This leads to the
>>>> distribution being mostly static, i.e. the user should not manually
>>>> install
>>>> / uninstall stuff.
>>>> A demo is available at [8]
>>>> Based on this idea, I improved the maven plugin a bit to allow
>> generating
>>>> mostly static distribution : the profiles / features are all installed
>> in
>>>> etc/startup.properties. A simplistic read-only configuration admin is
>>>> used
>>>> that will pick up configuration in the etc/ folder directly. This lead
>> to
>>>> no runtime dependencies for the provisioning of karaf itself :
>>>> fileinstall,
>>>> the features service can all be removed. The bare framework can only
>>>> contain 3 bundles : pax-logging + the static configadmin [9]. One
>>>> improvement is that the distribution is generated using reference:file:
>>>> urls in startup.properties, avoiding a copy of each jar into the cache.
>>>> An
>>>> additional improvement could be to pre-compute the wiring which can take
>>>> some time (but this is not supported by felix).
>>>>
>>>>
>>>> Thoughts welcomed !
>>>>
>>>>
>>>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
>>>> [2]
>>>> https://github.com/apache/karaf/blob/master/demos/
>>>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
>>>> [3]
>>>> https://github.com/apache/karaf/blob/master/demos/
>>>> profiles/registry/src/main/resources/default.profile/version.cfg
>>>> [4]
>>>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
>>>> fsp/zipfilesystemprovider.html
>>>> [5] https://github.com/gnodet/githubfs
>>>> [6]
>>>>
>> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
>>>> e16f618704
>>>> [7] https://github.com/google/jimfs
>>>> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
>>>> [9]
>>>> https://github.com/apache/karaf/blob/master/demos/
>>>> profiles/static/src/main/resources/features.xml
>>>>
>>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>>
>> --
>>
>> Apache Member
>> Apache Karaf <http://karaf.apache.org/> Committer & PMC
>> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
>> Project Lead
>> blog <http://notizblog.nierbeck.de/>
>> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
>>
>> Software Architect / Project Manager / Scrum Master
>>
--
Krzysztof Sobkowiak
JEE & OSS Architect | Senior Solution Architect @ Capgemini | Committer
@ ASF
Capgemini <http://www.pl.capgemini.com/> | Software Solutions Center
<http://www.pl.capgemini-sdm.com/> | Wroclaw
e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
Twitter: @KSobkowiak
Calendar: http://goo.gl/yvsebC
Re: [HEADS UP] Profiles and static distributions
Posted by Guillaume Nodet <gn...@apache.org>.
Profiles are only used to generate a new assembly or child instance.
The "static" distributions is related to profiles, but can be generated out
of a pure feature list. Such distributions, once started, are usually
limited (my initial goal was to really lock it down), at least that's the
case in the demo i pushed. But this is somewhat controlled by the demo
itself (i.e. the fact that file install + features service aren't installed
is because the plugin is told to use a non standard framework instead of
the default one). So once it has been started, such a static distribution
is "mostly" static, but it's still a valid osgi framework, so if you have a
way to access it (through the locale console, ssh or jmx) you could
manually install the features service bundle, then use it install
additional features. But if everything has been disabled ...
That said, I suppose profiles could be enhanced to support applying a
profile at runtime, but it opens a whole bunch of different problems.
2014-12-11 22:46 GMT+01:00 Achim Nierbeck <bc...@googlemail.com>:
> Hi Guillaume,
>
> I'd have to play with it to get the full picture of the consequences.
> But from what I've seen so fare I like the idea.
> Just for clarification, profiles are a "blueprint" for generating a custom
> distribution during build time, but not while running already
> or is it possible to start a "static" karaf container and customize it
> after it has been started?
>
> regards, Achim
>
> 2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
>
> > Thanks Guillaume,
> >
> > I gonna review and update the documentation.
> >
> > Regards
> > JB
> >
> >
> > On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
> >
> >> I've just committed to the 4.x branch two new features:
> >> * profiles
> >> * static distributions
> >>
> >> Profiles
> >> ======
> >>
> >> A profile is a data structure that brings together configuration and
> >> provisioning, so it's related to features, but is quite different. A
> demo
> >> is available at [1].
> >> A profile contains a list of configuration files which can be either
> >> properties based or in any other format. Properties based
> configurations
> >> are handled in a very specific way, as they can overlay each other (more
> >> below).
> >> One of this properties based configuration holds provisioning
> informations
> >> along with profiles specific attributes, such as a list of parent
> >> profiles,
> >> a list of bundles, a list of features repositories and a list of
> features.
> >>
> >> A profile can have one or more parents, so the hierarchy is an inverted
> >> tree. This allows to define generic profiles and specialise them in
> >> children.
> >>
> >> Overlay profile: at some point, there is a need to flatten this
> hierarchy
> >> into an "overlay" profile. This process is done by walking into the
> >> parents hierarchy (depth first) and computing the resulting
> configuration
> >> for each configuration file. Non properties configuration files simply
> >> overwrite each other, while properties based configuration complement
> each
> >> other (with the ability to clear a single value or all values).
> >>
> >> Effective profile: properties configurations can contain placeholders to
> >> resolve. The only one defined atm is a property placeholder pointing to
> a
> >> value in a different configuration. Those placeholders are resolved on
> >> the
> >> overlay profile to give the end result effective profile which will be
> >> actually used. An example of using this placeholder is shown in [2]
> which
> >> end up pointing to the configuration in its parent [3]
> >>
> >> Provisioning: each profile contains a list of feature repositories,
> >> features and bundles. Those information will be used in the effective
> >> profile to know which bundles and features are needed on a given
> instance.
> >> Profiles are not used at runtime for the moment and there are 2 main
> >> usage : creating child instances and generating distributions using the
> >> maven plugin.
> >>
> >> Profile manipulation : profiles are manipulated using the Profiles class
> >> (static helpers) and the ProfileBuilder (to build profile instances,
> which
> >> are immutable). Reading and writing profiles is done using the jdk7 in
> >> FileSystem / Path interface, which obviously provides support for the
> >> standard file system, but could be used with various file systems
> (inside
> >> a
> >> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
> >> Those are the ones I know about). A set of commands can be used to edit
> >> the profiles.
> >>
> >> Further considerations: possible improvements include an overlay
> mechanism
> >> for xml, integration with cellar, file systems using a real git repo,
> >> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
> >> too. Better integration with kars could be provided too.
> >>
> >>
> >> Static distributions
> >> ==============
> >>
> >> One use case of profiles is to generate distributions. Changes on the
> >> profile need a rebuild on the distribution (this may or may not be a
> >> problem, depending on your environment, but if it can seems weird in an
> >> OSGi world, it's not so much in a cloud environment). This leads to the
> >> distribution being mostly static, i.e. the user should not manually
> >> install
> >> / uninstall stuff.
> >> A demo is available at [8]
> >> Based on this idea, I improved the maven plugin a bit to allow
> generating
> >> mostly static distribution : the profiles / features are all installed
> in
> >> etc/startup.properties. A simplistic read-only configuration admin is
> >> used
> >> that will pick up configuration in the etc/ folder directly. This lead
> to
> >> no runtime dependencies for the provisioning of karaf itself :
> >> fileinstall,
> >> the features service can all be removed. The bare framework can only
> >> contain 3 bundles : pax-logging + the static configadmin [9]. One
> >> improvement is that the distribution is generated using reference:file:
> >> urls in startup.properties, avoiding a copy of each jar into the cache.
> >> An
> >> additional improvement could be to pre-compute the wiring which can take
> >> some time (but this is not supported by felix).
> >>
> >>
> >> Thoughts welcomed !
> >>
> >>
> >> [1] https://github.com/apache/karaf/tree/master/demos/profiles
> >> [2]
> >> https://github.com/apache/karaf/blob/master/demos/
> >> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
> >> [3]
> >> https://github.com/apache/karaf/blob/master/demos/
> >> profiles/registry/src/main/resources/default.profile/version.cfg
> >> [4]
> >> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
> >> fsp/zipfilesystemprovider.html
> >> [5] https://github.com/gnodet/githubfs
> >> [6]
> >>
> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
> >> e16f618704
> >> [7] https://github.com/google/jimfs
> >> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
> >> [9]
> >> https://github.com/apache/karaf/blob/master/demos/
> >> profiles/static/src/main/resources/features.xml
> >>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
>
>
> --
>
> Apache Member
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
> Project Lead
> blog <http://notizblog.nierbeck.de/>
> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
>
> Software Architect / Project Manager / Scrum Master
>
Re: [HEADS UP] Profiles and static distributions
Posted by Achim Nierbeck <bc...@googlemail.com>.
Hi Guillaume,
I'd have to play with it to get the full picture of the consequences.
But from what I've seen so fare I like the idea.
Just for clarification, profiles are a "blueprint" for generating a custom
distribution during build time, but not while running already
or is it possible to start a "static" karaf container and customize it
after it has been started?
regards, Achim
2014-12-11 10:19 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
> Thanks Guillaume,
>
> I gonna review and update the documentation.
>
> Regards
> JB
>
>
> On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
>
>> I've just committed to the 4.x branch two new features:
>> * profiles
>> * static distributions
>>
>> Profiles
>> ======
>>
>> A profile is a data structure that brings together configuration and
>> provisioning, so it's related to features, but is quite different. A demo
>> is available at [1].
>> A profile contains a list of configuration files which can be either
>> properties based or in any other format. Properties based configurations
>> are handled in a very specific way, as they can overlay each other (more
>> below).
>> One of this properties based configuration holds provisioning informations
>> along with profiles specific attributes, such as a list of parent
>> profiles,
>> a list of bundles, a list of features repositories and a list of features.
>>
>> A profile can have one or more parents, so the hierarchy is an inverted
>> tree. This allows to define generic profiles and specialise them in
>> children.
>>
>> Overlay profile: at some point, there is a need to flatten this hierarchy
>> into an "overlay" profile. This process is done by walking into the
>> parents hierarchy (depth first) and computing the resulting configuration
>> for each configuration file. Non properties configuration files simply
>> overwrite each other, while properties based configuration complement each
>> other (with the ability to clear a single value or all values).
>>
>> Effective profile: properties configurations can contain placeholders to
>> resolve. The only one defined atm is a property placeholder pointing to a
>> value in a different configuration. Those placeholders are resolved on
>> the
>> overlay profile to give the end result effective profile which will be
>> actually used. An example of using this placeholder is shown in [2] which
>> end up pointing to the configuration in its parent [3]
>>
>> Provisioning: each profile contains a list of feature repositories,
>> features and bundles. Those information will be used in the effective
>> profile to know which bundles and features are needed on a given instance.
>> Profiles are not used at runtime for the moment and there are 2 main
>> usage : creating child instances and generating distributions using the
>> maven plugin.
>>
>> Profile manipulation : profiles are manipulated using the Profiles class
>> (static helpers) and the ProfileBuilder (to build profile instances, which
>> are immutable). Reading and writing profiles is done using the jdk7 in
>> FileSystem / Path interface, which obviously provides support for the
>> standard file system, but could be used with various file systems (inside
>> a
>> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
>> Those are the ones I know about). A set of commands can be used to edit
>> the profiles.
>>
>> Further considerations: possible improvements include an overlay mechanism
>> for xml, integration with cellar, file systems using a real git repo,
>> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
>> too. Better integration with kars could be provided too.
>>
>>
>> Static distributions
>> ==============
>>
>> One use case of profiles is to generate distributions. Changes on the
>> profile need a rebuild on the distribution (this may or may not be a
>> problem, depending on your environment, but if it can seems weird in an
>> OSGi world, it's not so much in a cloud environment). This leads to the
>> distribution being mostly static, i.e. the user should not manually
>> install
>> / uninstall stuff.
>> A demo is available at [8]
>> Based on this idea, I improved the maven plugin a bit to allow generating
>> mostly static distribution : the profiles / features are all installed in
>> etc/startup.properties. A simplistic read-only configuration admin is
>> used
>> that will pick up configuration in the etc/ folder directly. This lead to
>> no runtime dependencies for the provisioning of karaf itself :
>> fileinstall,
>> the features service can all be removed. The bare framework can only
>> contain 3 bundles : pax-logging + the static configadmin [9]. One
>> improvement is that the distribution is generated using reference:file:
>> urls in startup.properties, avoiding a copy of each jar into the cache.
>> An
>> additional improvement could be to pre-compute the wiring which can take
>> some time (but this is not supported by felix).
>>
>>
>> Thoughts welcomed !
>>
>>
>> [1] https://github.com/apache/karaf/tree/master/demos/profiles
>> [2]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
>> [3]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/registry/src/main/resources/default.profile/version.cfg
>> [4]
>> http://docs.oracle.com/javase/7/docs/technotes/guides/io/
>> fsp/zipfilesystemprovider.html
>> [5] https://github.com/gnodet/githubfs
>> [6]
>> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09
>> e16f618704
>> [7] https://github.com/google/jimfs
>> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
>> [9]
>> https://github.com/apache/karaf/blob/master/demos/
>> profiles/static/src/main/resources/features.xml
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Apache Member
Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
Project Lead
blog <http://notizblog.nierbeck.de/>
Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS>
Software Architect / Project Manager / Scrum Master
Re: [HEADS UP] Profiles and static distributions
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks Guillaume,
I gonna review and update the documentation.
Regards
JB
On 12/11/2014 09:04 AM, Guillaume Nodet wrote:
> I've just committed to the 4.x branch two new features:
> * profiles
> * static distributions
>
> Profiles
> ======
>
> A profile is a data structure that brings together configuration and
> provisioning, so it's related to features, but is quite different. A demo
> is available at [1].
> A profile contains a list of configuration files which can be either
> properties based or in any other format. Properties based configurations
> are handled in a very specific way, as they can overlay each other (more
> below).
> One of this properties based configuration holds provisioning informations
> along with profiles specific attributes, such as a list of parent profiles,
> a list of bundles, a list of features repositories and a list of features.
>
> A profile can have one or more parents, so the hierarchy is an inverted
> tree. This allows to define generic profiles and specialise them in
> children.
>
> Overlay profile: at some point, there is a need to flatten this hierarchy
> into an "overlay" profile. This process is done by walking into the
> parents hierarchy (depth first) and computing the resulting configuration
> for each configuration file. Non properties configuration files simply
> overwrite each other, while properties based configuration complement each
> other (with the ability to clear a single value or all values).
>
> Effective profile: properties configurations can contain placeholders to
> resolve. The only one defined atm is a property placeholder pointing to a
> value in a different configuration. Those placeholders are resolved on the
> overlay profile to give the end result effective profile which will be
> actually used. An example of using this placeholder is shown in [2] which
> end up pointing to the configuration in its parent [3]
>
> Provisioning: each profile contains a list of feature repositories,
> features and bundles. Those information will be used in the effective
> profile to know which bundles and features are needed on a given instance.
> Profiles are not used at runtime for the moment and there are 2 main
> usage : creating child instances and generating distributions using the
> maven plugin.
>
> Profile manipulation : profiles are manipulated using the Profiles class
> (static helpers) and the ProfileBuilder (to build profile instances, which
> are immutable). Reading and writing profiles is done using the jdk7 in
> FileSystem / Path interface, which obviously provides support for the
> standard file system, but could be used with various file systems (inside a
> zip [4], readonly github remote repository [5], sftp [6], in memory [7].
> Those are the ones I know about). A set of commands can be used to edit
> the profiles.
>
> Further considerations: possible improvements include an overlay mechanism
> for xml, integration with cellar, file systems using a real git repo,
> hazelcast, zookeeper, etc... Applying profiles at runtime could be done
> too. Better integration with kars could be provided too.
>
>
> Static distributions
> ==============
>
> One use case of profiles is to generate distributions. Changes on the
> profile need a rebuild on the distribution (this may or may not be a
> problem, depending on your environment, but if it can seems weird in an
> OSGi world, it's not so much in a cloud environment). This leads to the
> distribution being mostly static, i.e. the user should not manually install
> / uninstall stuff.
> A demo is available at [8]
> Based on this idea, I improved the maven plugin a bit to allow generating
> mostly static distribution : the profiles / features are all installed in
> etc/startup.properties. A simplistic read-only configuration admin is used
> that will pick up configuration in the etc/ folder directly. This lead to
> no runtime dependencies for the provisioning of karaf itself : fileinstall,
> the features service can all be removed. The bare framework can only
> contain 3 bundles : pax-logging + the static configadmin [9]. One
> improvement is that the distribution is generated using reference:file:
> urls in startup.properties, avoiding a copy of each jar into the cache. An
> additional improvement could be to pre-compute the wiring which can take
> some time (but this is not supported by felix).
>
>
> Thoughts welcomed !
>
>
> [1] https://github.com/apache/karaf/tree/master/demos/profiles
> [2]
> https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/karaf.profile/profile.cfg#L25
> [3]
> https://github.com/apache/karaf/blob/master/demos/profiles/registry/src/main/resources/default.profile/version.cfg
> [4]
> http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html
> [5] https://github.com/gnodet/githubfs
> [6]
> https://github.com/gnodet/mina-sshd/commit/f68468c686099f6f12a9093e949c09e16f618704
> [7] https://github.com/google/jimfs
> [8] https://github.com/apache/karaf/blob/master/demos/profiles/static/
> [9]
> https://github.com/apache/karaf/blob/master/demos/profiles/static/src/main/resources/features.xml
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com