Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

Hi Konrad & all,

sorry for being radio silent for almost half a year now but as you might
have noticed I was quite busy in my dayjob and the stuff I committed before
did just work and priorities didn't allow me to get back to the whole
content-package stuff (which I hope to slowly change now).

Based on the last changes in the filevault plugin some of the setups that
were in place are no longer working - this is specifically by enforcing to
not embed osgi bundles and configs.

My suggestion to make sure this is not breaking each and every existing
setup would be 2fold
a) make sure we can override those rules to run it in some kind of legacy
mode
b) we optimize the tooling to support a merging of container packages into
one container or provide a nesting capability

As an endresult it should be possible to slowly migrate existing structures
(that currently are postprocessed by a converter) towards something that
natively enables a suitable structure while the codestructures would not
need to change beyond the pom definitions.

Right now in most cases because of compatiblity reasons with the osgi
installer the osgi configs and osgi bundles are part of the application
packages. This is failing for almost everyone - and if we want them to be
moved out into the container we must make sure that the structures can be
reflected.

As a first step I would suggest we establish some property to establish
some legacy mode for application packages and as a next step make sure we
get container packages that allow the same that is currently partially
reflected by application packages with mutliple install folders (e.g. with
runmode support).

As long as we want to support the sling installer for a while the container
structure itself would need be able to locate the binaries and packages in
locations the installer is aware of (install and config folders) or we need
to patch the sling installer to be able to process the new container
packages accurately

Cheers
Dominik

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

I created https://issues.apache.org/jira/browse/JCRVLT-401 <https://issues.apache.org/jira/browse/JCRVLT-401> for a clarification of allowed contents of a container package.

> On 21. Jan 2020, at 16:07, Dominik Süß <do...@gmail.com> wrote:
> 
> Yes I get that perspective and I was a bit surprised to read it like that - I assume when toby wrote it down the need for mutable content as seeding content wasn't really present (or an alternative approach discussed like in sling with repoinit - yet we still have the need to support those for a while).
> The question is now if relaxing the definition would be the right way to go -  I guess we don#t have any logic depending on those constraints right now beside of the analysers failing on those. So iterating over the definition might be the right thing to do.
> 
> Not sure if Toby has a specific opinion on that one.
> 
> Cheers
> Dominik
> 
> 
> 
> On Tue, Jan 21, 2020 at 3:40 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
> That is not what is stated in JCRVLT-170.
> It is a bit tricky though because JCRVLT-170 combines blacklisting and whitelisting rules and doesn't say what the baseline is.
> ...
> Container Packages
> As for the container / deployment packages they have the characteristics that:
> they don't contain any content
> they contain a set of application content packages
> they don't have dependencies on other deployment packages (i.e. the dependencies are implicit given by the relations of the included artifacts)
> 
> ....
> 
> I think everyone would derive from this info that "content" type content package are not allowed, because only "application" content packages are mentioned here.
> But I am fine with a clarification, that content packages may be contained in there as well. For that we should have a new FileVault ticket!
> 
> Then we don't necessarily need to have another package type. But we should then also clarify if nesting of container packages should be allowed or not.
> IMHO this would be a useful feature and I don't see why we should disallow that.
> 
> Konrad
> 
> 
>> On 21. Jan 2020, at 15:21, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>> 
>> well TBH - the original understanding I had about container was what I was describing with the collection with the difference that it has an identity (yet it isn't defined anywhere and not declared that you can have dependencies on containers. Core concept that we discussed for JCRVLT-170 is that the containers would never directly install anything in the repository directly but is a container for packages and other installables as well as meta that may be installed by what ever processes them. This is the origin of not having "any" content in them - basically, anything that is to be installed would have an own identity so you can declare to any content or installable artifact by referencing their specific identity.
>> 
>> So if we refine the definition of container and relax it what is currently enforced with the analyzers today we would be able to follow this deployment scheme and are able to enforce to not base on antipatterns purely by blocking mixed packages for scenarios with composite  nodestore (btw. being able to configure the analysers to enforce certain packageTypes would be really useful for that).
>> 
>> WDYT?
>> 
>> Cheers
>> Dominik
>> 
>> 
>> 
>> On Tue, Jan 21, 2020 at 1:50 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>> Hi Dominik,
>> This is still very vague to me. 
>> 
>> What is your idea about the current packageType: container (which is a content-package with an identity)? What should be allowed in there?
>> What about the new propsed type PackageType.Collection? How does that relate to the existing packageType? Does it replace it?
>> 
>> Can you try to answer those questions, so that everyone can get a clearer picture at what you are aiming?
>> 
>> I would strongly suggest not to come up with something which is not a content-package itself, because it seems no one has the time to implement new features in FileVault (like. a package which can only be unwrapped, but no one can depend on).
>> Let's rather enforce this with some validators: https://jackrabbit.apache.org/filevault/validation.html <https://jackrabbit.apache.org/filevault/validation.html>. Way less effort IMHO and much better backwards compatible!
>> Thanks,
>> Konrad
>> 
>> 
>>> On 21. Jan 2020, at 09:20, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Hi Konrad,
>>> 
>>> what worries me about mixed is that is doesn't enforce any separation which is the scenario that we want to prevent. 
>>> A mixed content-package has an identity which another package can depend on - yet in a composite nodestore the mutable and immutable parts are installed at different times making it impossible to model the dependencies of immutable packages against immutable content tha would be served from a mixed package.
>>> This is why supporting mixed content packages cannot work in a composite nodestore scenario.
>>> 
>>> Yes it is possible to abuse mixed packages as what I propose as a collection type - yet no one prevents you from adding content directly to that package which in turn might require to set direct dependencies for it making it again something you cannot simply drop
>>> 
>>> Therefore the suggestion is to have a type that allows bundling together everything you want to eventually get installed into the system, where all packages that have an identity are clearly either application or content (so separation of mutable and immutable) so the dependency ordering is no longer ambiguous between deployment (change of immutable content state) and runtime (change of mutable state).
>>> 
>>> I understand that the perspective you have is from what is the state when you would upload this to packagemanager - and from my pov this would only alow you to unfold that package so you have all the contained packages available. And yes we might even add a ui / rest api feature to triggere installation for all of the contained packages automatically (e.g. the mutable subset or all if there is no CNS) yet the package itself would not have an installation state on its own as it is only a shipping container.
>>> 
>>> Does this clarify my thoughts about this a bit better?
>>> 
>>> When it comes to nesting of packages of same type it is ambiguous so we might want to clarify this for sure.
>>> 
>>> Cheers
>>> Dominik
>>> 
>>> On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>> Hi Dominik,
>>> I am not getting exactly what you propose here:
>>> Which types should be allowed in "container"?
>>> a) If there is already a support for "content" type packages I don't see the need for another type within "container"
>>> b) If there is no support for nested content type packages, the question is how to aggregate multiple content type packages together so that they are actually installed via one big content-package?
>>> 
>>> As I already said, I am not too much worried about a mixture of mutable and immutable packages (that can still be done via 'mixed') but if we support that in FileVault via a dedicated type, this should be installable as well (i.e. be a regular content package).
>>> Having a pure package format without supporting installation sounds like a unnecessary limitation (even having AEM as a Cloud Service in mind) because a lot of consumers right now don't care about mutable/immutable (i.e. old AEM instances not leveraging the Composite Node Store).
>>> As I already said, I would like to see a clarification ticket for JCRVLT-170, because that clearly does neither mention nested containers nor content type packages within containers (therefore everyone assumes, those are forbidden)
>>> 
>>> Thanks,
>>> Konrad
>>> 
>>>> On 21. Jan 2020, at 08:34, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>> 
>>>> Hi Konrad,
>>>> 
>>>> I get your point yet that was not really the origin. Probably we can split this off in another way.
>>>> Let's introduce a PackageType.Collection - basically this would supposed to be never installed but is just a shipping unit holding a set of other packages. These could either be in a flat list or for compat reasons also be  placed in an /etc/packages structure.
>>>> This packageType would be excluded of evaluation and is supposed to never be installed directly. It basically is a zip file acting as a wrapper around a collection of other packages.
>>>> No further semantics beside the fact that it has no vault identity on its own and won't be visible after being installed - it's a pure shipping unit.
>>>> 
>>>> Cheers
>>>> Dominik
>>>> 
>>>> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>>> Hi Dominik,
>>>> For me container was always immutable by only embedding immutable content (bundles, config, app-packages, as indicated in JCRVLT-170)
>>>> Especially when we allow nesting of containers, we should probably think of two different container types:
>>>> - container (immutable), with the current restrictions, but in addition allows nested containers (immutable)
>>>> - content-container (mutable), only allows content and content-container sub packages
>>>> 
>>>> That way we have a clean separation even on the container level.
>>>> Whether we need one container which allows embedding both container (immutable) and content-container (mutable), I am not sure about. WDYT? Also not very sure about the naming of such a construct...
>>>> 
>>>> Konrad
>>>> 
>>>>> On 16. Jan 2020, at 16:40, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>>> 
>>>>> Hi Konrad,
>>>>> 
>>>>> mixed is a legacy structure that basically allows all kinds of things. What we would want to avoid is mixing content and application content in the same package - building a container that allow application and content package to be grouped together (as installable entities) but making sure they are not mashed together helps to install both at distinct times. If you check setups working with the composite nodestore you would have 2 different times at which the application and the content packages would be installed. Crucial is that containers basically don't really define an own identity that you would be depending on - you should only define dependencies on the actual installables . The problem of a mixed package is that people start to add content to it and therefore you have to define dependencies on it, if they then also contain application on content there is ambiguity when this package actually is in an installed state as the application part would need to be installed when preparing the immutable part of the repository while the content only can be installed at runtime with the mutable content being accessible.
>>>>> Containers would be the tool to declare a grouping of entities that are supposed to be installed while (optimally - not sure if we should enforce) not playing a role in the dependency handling at all.
>>>>> 
>>>>> Cheers
>>>>> Dominik
>>>>> 
>>>>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>>>> Hi Dominik,
>>>>> better late feedback than no feedback at all.
>>>>> Let me comment on your points
>>>>> 
>>>>>> On 16. Jan 2020, at 15:09, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>>>> 
>>>>>> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
>>>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>>>> 
>>>>>> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
>>>>>> a) make sure we can override those rules to run it in some kind of legacy mode
>>>>> That is implicitly done by setting the packageType to 'mixed'.
>>>>> But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?
>>>>> 
>>>>>> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
>>>>> There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
>>>>> What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
>>>>>> 
>>>>>> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
>>>>> With 'mixed' this is already the case IMHO, other changes might need further refactorings....
>>>>>> 
>>>>>> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
>>>>> Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
>>>>>> 
>>>>>> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
>>>>> What exactly is wrong about 'mixed'?
>>>>>> 
>>>>>> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
>>>>> Container packages allow exactly that IMHO
>>>>>> 
>>>>>> Cheers
>>>>>> Dominik
>>>>>> 
>>>>> Konrad
>>>> 
>>> 
>> 
> 

Re: Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

Yes I get that perspective and I was a bit surprised to read it like that -
I assume when toby wrote it down the need for mutable content as seeding
content wasn't really present (or an alternative approach discussed like in
sling with repoinit - yet we still have the need to support those for a
while).
The question is now if relaxing the definition would be the right way to go
-  I guess we don#t have any logic depending on those constraints right now
beside of the analysers failing on those. So iterating over the definition
might be the right thing to do.

Not sure if Toby has a specific opinion on that one.

Cheers
Dominik



On Tue, Jan 21, 2020 at 3:40 PM Konrad Windszus <ko...@gmx.de> wrote:

> That is not what is stated in JCRVLT-170.
> It is a bit tricky though because JCRVLT-170 combines blacklisting and
> whitelisting rules and doesn't say what the baseline is.
> ...
> Container Packages
>
> As for the container / deployment packages they have the characteristics
> that:
>
>    - they don't contain any content
>    - they contain a set of application content packages
>    - they don't have dependencies on other deployment packages (i.e. the
>    dependencies are implicit given by the relations of the included artifacts)
>
>
> ....
>
> I think everyone would derive from this info that "content" type content
> package are not allowed, because only "application" content packages are
> mentioned here.
> But I am fine with a clarification, that content packages may be contained
> in there as well. For that we should have a new FileVault ticket!
>
> Then we don't necessarily need to have another package type. But we should
> then also clarify if nesting of container packages should be allowed or not.
> IMHO this would be a useful feature and I don't see why we should disallow
> that.
>
> Konrad
>
>
> On 21. Jan 2020, at 15:21, Dominik Süß <do...@gmail.com> wrote:
>
> well TBH - the original understanding I had about container was what I was
> describing with the collection with the difference that it has an identity
> (yet it isn't defined anywhere and not declared that you can have
> dependencies on containers. Core concept that we discussed for JCRVLT-170
> is that the containers would never directly install anything in the
> repository directly but is a container for packages and other installables
> as well as meta that may be installed by what ever processes them. This is
> the origin of not having "any" content in them - basically, anything that
> is to be installed would have an own identity so you can declare to any
> content or installable artifact by referencing their specific identity.
>
> So if we refine the definition of container and relax it what is currently
> enforced with the analyzers today we would be able to follow this
> deployment scheme and are able to enforce to not base on antipatterns
> purely by blocking mixed packages for scenarios with composite  nodestore
> (btw. being able to configure the analysers to enforce certain packageTypes
> would be really useful for that).
>
> WDYT?
>
> Cheers
> Dominik
>
>
>
> On Tue, Jan 21, 2020 at 1:50 PM Konrad Windszus <ko...@gmx.de> wrote:
>
>> Hi Dominik,
>> This is still very vague to me.
>>
>> What is your idea about the current packageType: container (which is a
>> content-package with an identity)? What should be allowed in there?
>> What about the new propsed type PackageType.Collection? How does that
>> relate to the existing packageType? Does it replace it?
>>
>> Can you try to answer those questions, so that everyone can get a clearer
>> picture at what you are aiming?
>>
>> I would strongly suggest not to come up with something which is not a
>> content-package itself, because it seems no one has the time to implement
>> new features in FileVault (like. a package which can only be unwrapped, but
>> no one can depend on).
>> Let's rather enforce this with some validators:
>> https://jackrabbit.apache.org/filevault/validation.html. Way less effort
>> IMHO and much better backwards compatible!
>> Thanks,
>> Konrad
>>
>>
>> On 21. Jan 2020, at 09:20, Dominik Süß <do...@gmail.com> wrote:
>>
>> Hi Konrad,
>>
>> what worries me about mixed is that is doesn't enforce any separation
>> which is the scenario that we want to prevent.
>> A mixed content-package has an identity which another package can depend
>> on - yet in a composite nodestore the mutable and immutable parts are
>> installed at different times making it impossible to model the dependencies
>> of immutable packages against immutable content tha would be served from a
>> mixed package.
>> This is why supporting mixed content packages cannot work in a composite
>> nodestore scenario.
>>
>> Yes it is possible to abuse mixed packages as what I propose as a
>> collection type - yet no one prevents you from adding content directly to
>> that package which in turn might require to set direct dependencies for it
>> making it again something you cannot simply drop
>>
>> Therefore the suggestion is to have a type that allows bundling together
>> everything you want to eventually get installed into the system, where all
>> packages that have an identity are clearly either application or content
>> (so separation of mutable and immutable) so the dependency ordering is no
>> longer ambiguous between deployment (change of immutable content state) and
>> runtime (change of mutable state).
>>
>> I understand that the perspective you have is from what is the state when
>> you would upload this to packagemanager - and from my pov this would only
>> alow you to unfold that package so you have all the contained packages
>> available. And yes we might even add a ui / rest api feature to triggere
>> installation for all of the contained packages automatically (e.g. the
>> mutable subset or all if there is no CNS) yet the package itself would not
>> have an installation state on its own as it is only a shipping container.
>>
>> Does this clarify my thoughts about this a bit better?
>>
>> When it comes to nesting of packages of same type it is ambiguous so we
>> might want to clarify this for sure.
>>
>> Cheers
>> Dominik
>>
>> On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <ko...@gmx.de> wrote:
>>
>>> Hi Dominik,
>>> I am not getting exactly what you propose here:
>>> Which types should be allowed in "container"?
>>> a) If there is already a support for "content" type packages I don't see
>>> the need for another type within "container"
>>> b) If there is no support for nested content type packages, the question
>>> is how to aggregate multiple content type packages together so that they
>>> are actually installed via one big content-package?
>>>
>>> As I already said, I am not too much worried about a mixture of mutable
>>> and immutable packages (that can still be done via 'mixed') but if we
>>> support that in FileVault via a dedicated type, this should be installable
>>> as well (i.e. be a regular content package).
>>> Having a pure package format without supporting installation sounds like
>>> a unnecessary limitation (even having AEM as a Cloud Service in mind)
>>> because a lot of consumers right now don't care about mutable/immutable
>>> (i.e. old AEM instances not leveraging the Composite Node Store).
>>> As I already said, I would like to see a clarification ticket for
>>> JCRVLT-170, because that clearly does neither mention nested containers nor
>>> content type packages within containers (therefore everyone assumes, those
>>> are forbidden)
>>>
>>> Thanks,
>>> Konrad
>>>
>>> On 21. Jan 2020, at 08:34, Dominik Süß <do...@gmail.com> wrote:
>>>
>>> Hi Konrad,
>>>
>>> I get your point yet that was not really the origin. Probably we can
>>> split this off in another way.
>>> Let's introduce a PackageType.Collection - basically this would supposed
>>> to be never installed but is just a shipping unit holding a set of other
>>> packages. These could either be in a flat list or for compat reasons also
>>> be  placed in an /etc/packages structure.
>>> This packageType would be excluded of evaluation and is supposed to
>>> never be installed directly. It basically is a zip file acting as a wrapper
>>> around a collection of other packages.
>>> No further semantics beside the fact that it has no vault identity on
>>> its own and won't be visible after being installed - it's a pure shipping
>>> unit.
>>>
>>> Cheers
>>> Dominik
>>>
>>> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <ko...@gmx.de> wrote:
>>>
>>>> Hi Dominik,
>>>> For me container was always immutable by only embedding immutable
>>>> content (bundles, config, app-packages, as indicated in JCRVLT-170)
>>>> Especially when we allow nesting of containers, we should probably
>>>> think of two different container types:
>>>> - container (immutable), with the current restrictions, but in addition
>>>> allows nested containers (immutable)
>>>> - content-container (mutable), only allows content and
>>>> content-container sub packages
>>>>
>>>> That way we have a clean separation even on the container level.
>>>> Whether we need one container which allows embedding both container
>>>> (immutable) and content-container (mutable), I am not sure about. WDYT?
>>>> Also not very sure about the naming of such a construct...
>>>>
>>>> Konrad
>>>>
>>>> On 16. Jan 2020, at 16:40, Dominik Süß <do...@gmail.com> wrote:
>>>>
>>>> Hi Konrad,
>>>>
>>>> mixed is a legacy structure that basically allows all kinds of things.
>>>> What we would want to avoid is mixing content and application content in
>>>> the same package - building a container that allow application and content
>>>> package to be grouped together (as installable entities) but making sure
>>>> they are not mashed together helps to install both at distinct times. If
>>>> you check setups working with the composite nodestore you would have 2
>>>> different times at which the application and the content packages would be
>>>> installed. Crucial is that containers basically don't really define an own
>>>> identity that you would be depending on - you should only define
>>>> dependencies on the actual installables . The problem of a mixed package is
>>>> that people start to add content to it and therefore you have to define
>>>> dependencies on it, if they then also contain application on content there
>>>> is ambiguity when this package actually is in an installed state as the
>>>> application part would need to be installed when preparing the immutable
>>>> part of the repository while the content only can be installed at runtime
>>>> with the mutable content being accessible.
>>>> Containers would be the tool to declare a grouping of entities that are
>>>> supposed to be installed while (optimally - not sure if we should enforce)
>>>> not playing a role in the dependency handling at all.
>>>>
>>>> Cheers
>>>> Dominik
>>>>
>>>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <ko...@gmx.de>
>>>> wrote:
>>>>
>>>>> Hi Dominik,
>>>>> better late feedback than no feedback at all.
>>>>> Let me comment on your points
>>>>>
>>>>> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Based on the last changes in the filevault plugin some of the setups
>>>>> that were in place are no longer working - this is specifically by
>>>>> enforcing to not embed osgi bundles and configs.
>>>>>
>>>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>>>
>>>>>
>>>>> My suggestion to make sure this is not breaking each and
>>>>> every existing setup would be 2fold
>>>>> a) make sure we can override those rules to run it in some kind of
>>>>> legacy mode
>>>>>
>>>>> That is implicitly done by setting the packageType to 'mixed'.
>>>>> But maybe I just don't understand correctly. I would help a lot to
>>>>> discuss on a concrete example. Can you share one which is no longer working?
>>>>>
>>>>> b) we optimize the tooling to support a merging of container packages
>>>>> into one container or provide a nesting capability
>>>>>
>>>>> There was a question just raised today on what a container is supposed
>>>>> to support. According to
>>>>> https://issues.apache.org/jira/browse/JCRVLT-170 neither nested
>>>>> container packages nor content packages.
>>>>> What exactly do you propose? Can you formulate your proposal in a
>>>>> dedicated JIRA ticket for FileVault?
>>>>>
>>>>>
>>>>> As an endresult it should be possible to slowly migrate existing
>>>>> structures (that currently are postprocessed by a converter) towards
>>>>> something that natively enables a suitable structure while the
>>>>> codestructures would not need to change beyond the pom definitions.
>>>>>
>>>>> With 'mixed' this is already the case IMHO, other changes might need
>>>>> further refactorings....
>>>>>
>>>>>
>>>>> Right now in most cases because of compatiblity reasons with the osgi
>>>>> installer the osgi configs and osgi bundles are part of the application
>>>>> packages. This is failing for almost everyone - and if we want them to be
>>>>> moved out into the container we must make sure that the structures can be
>>>>> reflected.
>>>>>
>>>>> Not necessarily. Even the OSGi installer can deal with container
>>>>> packages containing both app subpackages and bundles/configs.
>>>>>
>>>>>
>>>>> As a first step I would suggest we establish some property to
>>>>> establish some legacy mode for application packages and as a next step make
>>>>> sure we get container packages that allow the same that is currently
>>>>> partially reflected by application packages with mutliple install folders
>>>>> (e.g. with runmode support).
>>>>>
>>>>> What exactly is wrong about 'mixed'?
>>>>>
>>>>>
>>>>> As long as we want to support the sling installer for a while the
>>>>> container structure itself would need be able to locate the binaries and
>>>>> packages in locations the installer is aware of (install and config
>>>>> folders) or we need to patch the sling installer to be able to process the
>>>>> new container packages accurately
>>>>>
>>>>> Container packages allow exactly that IMHO
>>>>>
>>>>>
>>>>> Cheers
>>>>> Dominik
>>>>>
>>>>> Konrad
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

That is not what is stated in JCRVLT-170.
It is a bit tricky though because JCRVLT-170 combines blacklisting and whitelisting rules and doesn't say what the baseline is.
...
Container Packages
As for the container / deployment packages they have the characteristics that:
they don't contain any content
they contain a set of application content packages
they don't have dependencies on other deployment packages (i.e. the dependencies are implicit given by the relations of the included artifacts)

....

I think everyone would derive from this info that "content" type content package are not allowed, because only "application" content packages are mentioned here.
But I am fine with a clarification, that content packages may be contained in there as well. For that we should have a new FileVault ticket!

Then we don't necessarily need to have another package type. But we should then also clarify if nesting of container packages should be allowed or not.
IMHO this would be a useful feature and I don't see why we should disallow that.

Konrad


> On 21. Jan 2020, at 15:21, Dominik Süß <do...@gmail.com> wrote:
> 
> well TBH - the original understanding I had about container was what I was describing with the collection with the difference that it has an identity (yet it isn't defined anywhere and not declared that you can have dependencies on containers. Core concept that we discussed for JCRVLT-170 is that the containers would never directly install anything in the repository directly but is a container for packages and other installables as well as meta that may be installed by what ever processes them. This is the origin of not having "any" content in them - basically, anything that is to be installed would have an own identity so you can declare to any content or installable artifact by referencing their specific identity.
> 
> So if we refine the definition of container and relax it what is currently enforced with the analyzers today we would be able to follow this deployment scheme and are able to enforce to not base on antipatterns purely by blocking mixed packages for scenarios with composite  nodestore (btw. being able to configure the analysers to enforce certain packageTypes would be really useful for that).
> 
> WDYT?
> 
> Cheers
> Dominik
> 
> 
> 
> On Tue, Jan 21, 2020 at 1:50 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
> Hi Dominik,
> This is still very vague to me. 
> 
> What is your idea about the current packageType: container (which is a content-package with an identity)? What should be allowed in there?
> What about the new propsed type PackageType.Collection? How does that relate to the existing packageType? Does it replace it?
> 
> Can you try to answer those questions, so that everyone can get a clearer picture at what you are aiming?
> 
> I would strongly suggest not to come up with something which is not a content-package itself, because it seems no one has the time to implement new features in FileVault (like. a package which can only be unwrapped, but no one can depend on).
> Let's rather enforce this with some validators: https://jackrabbit.apache.org/filevault/validation.html <https://jackrabbit.apache.org/filevault/validation.html>. Way less effort IMHO and much better backwards compatible!
> Thanks,
> Konrad
> 
> 
>> On 21. Jan 2020, at 09:20, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi Konrad,
>> 
>> what worries me about mixed is that is doesn't enforce any separation which is the scenario that we want to prevent. 
>> A mixed content-package has an identity which another package can depend on - yet in a composite nodestore the mutable and immutable parts are installed at different times making it impossible to model the dependencies of immutable packages against immutable content tha would be served from a mixed package.
>> This is why supporting mixed content packages cannot work in a composite nodestore scenario.
>> 
>> Yes it is possible to abuse mixed packages as what I propose as a collection type - yet no one prevents you from adding content directly to that package which in turn might require to set direct dependencies for it making it again something you cannot simply drop
>> 
>> Therefore the suggestion is to have a type that allows bundling together everything you want to eventually get installed into the system, where all packages that have an identity are clearly either application or content (so separation of mutable and immutable) so the dependency ordering is no longer ambiguous between deployment (change of immutable content state) and runtime (change of mutable state).
>> 
>> I understand that the perspective you have is from what is the state when you would upload this to packagemanager - and from my pov this would only alow you to unfold that package so you have all the contained packages available. And yes we might even add a ui / rest api feature to triggere installation for all of the contained packages automatically (e.g. the mutable subset or all if there is no CNS) yet the package itself would not have an installation state on its own as it is only a shipping container.
>> 
>> Does this clarify my thoughts about this a bit better?
>> 
>> When it comes to nesting of packages of same type it is ambiguous so we might want to clarify this for sure.
>> 
>> Cheers
>> Dominik
>> 
>> On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>> Hi Dominik,
>> I am not getting exactly what you propose here:
>> Which types should be allowed in "container"?
>> a) If there is already a support for "content" type packages I don't see the need for another type within "container"
>> b) If there is no support for nested content type packages, the question is how to aggregate multiple content type packages together so that they are actually installed via one big content-package?
>> 
>> As I already said, I am not too much worried about a mixture of mutable and immutable packages (that can still be done via 'mixed') but if we support that in FileVault via a dedicated type, this should be installable as well (i.e. be a regular content package).
>> Having a pure package format without supporting installation sounds like a unnecessary limitation (even having AEM as a Cloud Service in mind) because a lot of consumers right now don't care about mutable/immutable (i.e. old AEM instances not leveraging the Composite Node Store).
>> As I already said, I would like to see a clarification ticket for JCRVLT-170, because that clearly does neither mention nested containers nor content type packages within containers (therefore everyone assumes, those are forbidden)
>> 
>> Thanks,
>> Konrad
>> 
>>> On 21. Jan 2020, at 08:34, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Hi Konrad,
>>> 
>>> I get your point yet that was not really the origin. Probably we can split this off in another way.
>>> Let's introduce a PackageType.Collection - basically this would supposed to be never installed but is just a shipping unit holding a set of other packages. These could either be in a flat list or for compat reasons also be  placed in an /etc/packages structure.
>>> This packageType would be excluded of evaluation and is supposed to never be installed directly. It basically is a zip file acting as a wrapper around a collection of other packages.
>>> No further semantics beside the fact that it has no vault identity on its own and won't be visible after being installed - it's a pure shipping unit.
>>> 
>>> Cheers
>>> Dominik
>>> 
>>> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>> Hi Dominik,
>>> For me container was always immutable by only embedding immutable content (bundles, config, app-packages, as indicated in JCRVLT-170)
>>> Especially when we allow nesting of containers, we should probably think of two different container types:
>>> - container (immutable), with the current restrictions, but in addition allows nested containers (immutable)
>>> - content-container (mutable), only allows content and content-container sub packages
>>> 
>>> That way we have a clean separation even on the container level.
>>> Whether we need one container which allows embedding both container (immutable) and content-container (mutable), I am not sure about. WDYT? Also not very sure about the naming of such a construct...
>>> 
>>> Konrad
>>> 
>>>> On 16. Jan 2020, at 16:40, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>> 
>>>> Hi Konrad,
>>>> 
>>>> mixed is a legacy structure that basically allows all kinds of things. What we would want to avoid is mixing content and application content in the same package - building a container that allow application and content package to be grouped together (as installable entities) but making sure they are not mashed together helps to install both at distinct times. If you check setups working with the composite nodestore you would have 2 different times at which the application and the content packages would be installed. Crucial is that containers basically don't really define an own identity that you would be depending on - you should only define dependencies on the actual installables . The problem of a mixed package is that people start to add content to it and therefore you have to define dependencies on it, if they then also contain application on content there is ambiguity when this package actually is in an installed state as the application part would need to be installed when preparing the immutable part of the repository while the content only can be installed at runtime with the mutable content being accessible.
>>>> Containers would be the tool to declare a grouping of entities that are supposed to be installed while (optimally - not sure if we should enforce) not playing a role in the dependency handling at all.
>>>> 
>>>> Cheers
>>>> Dominik
>>>> 
>>>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>>> Hi Dominik,
>>>> better late feedback than no feedback at all.
>>>> Let me comment on your points
>>>> 
>>>>> On 16. Jan 2020, at 15:09, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>>> 
>>>>> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
>>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>>> 
>>>>> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
>>>>> a) make sure we can override those rules to run it in some kind of legacy mode
>>>> That is implicitly done by setting the packageType to 'mixed'.
>>>> But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?
>>>> 
>>>>> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
>>>> There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
>>>> What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
>>>>> 
>>>>> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
>>>> With 'mixed' this is already the case IMHO, other changes might need further refactorings....
>>>>> 
>>>>> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
>>>> Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
>>>>> 
>>>>> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
>>>> What exactly is wrong about 'mixed'?
>>>>> 
>>>>> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
>>>> Container packages allow exactly that IMHO
>>>>> 
>>>>> Cheers
>>>>> Dominik
>>>>> 
>>>> Konrad
>>> 
>> 
> 

Re: Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

well TBH - the original understanding I had about container was what I was
describing with the collection with the difference that it has an identity
(yet it isn't defined anywhere and not declared that you can have
dependencies on containers. Core concept that we discussed for JCRVLT-170
is that the containers would never directly install anything in the
repository directly but is a container for packages and other installables
as well as meta that may be installed by what ever processes them. This is
the origin of not having "any" content in them - basically, anything that
is to be installed would have an own identity so you can declare to any
content or installable artifact by referencing their specific identity.

So if we refine the definition of container and relax it what is currently
enforced with the analyzers today we would be able to follow this
deployment scheme and are able to enforce to not base on antipatterns
purely by blocking mixed packages for scenarios with composite  nodestore
(btw. being able to configure the analysers to enforce certain packageTypes
would be really useful for that).

WDYT?

Cheers
Dominik



On Tue, Jan 21, 2020 at 1:50 PM Konrad Windszus <ko...@gmx.de> wrote:

> Hi Dominik,
> This is still very vague to me.
>
> What is your idea about the current packageType: container (which is a
> content-package with an identity)? What should be allowed in there?
> What about the new propsed type PackageType.Collection? How does that
> relate to the existing packageType? Does it replace it?
>
> Can you try to answer those questions, so that everyone can get a clearer
> picture at what you are aiming?
>
> I would strongly suggest not to come up with something which is not a
> content-package itself, because it seems no one has the time to implement
> new features in FileVault (like. a package which can only be unwrapped, but
> no one can depend on).
> Let's rather enforce this with some validators:
> https://jackrabbit.apache.org/filevault/validation.html. Way less effort
> IMHO and much better backwards compatible!
> Thanks,
> Konrad
>
>
> On 21. Jan 2020, at 09:20, Dominik Süß <do...@gmail.com> wrote:
>
> Hi Konrad,
>
> what worries me about mixed is that is doesn't enforce any separation
> which is the scenario that we want to prevent.
> A mixed content-package has an identity which another package can depend
> on - yet in a composite nodestore the mutable and immutable parts are
> installed at different times making it impossible to model the dependencies
> of immutable packages against immutable content tha would be served from a
> mixed package.
> This is why supporting mixed content packages cannot work in a composite
> nodestore scenario.
>
> Yes it is possible to abuse mixed packages as what I propose as a
> collection type - yet no one prevents you from adding content directly to
> that package which in turn might require to set direct dependencies for it
> making it again something you cannot simply drop
>
> Therefore the suggestion is to have a type that allows bundling together
> everything you want to eventually get installed into the system, where all
> packages that have an identity are clearly either application or content
> (so separation of mutable and immutable) so the dependency ordering is no
> longer ambiguous between deployment (change of immutable content state) and
> runtime (change of mutable state).
>
> I understand that the perspective you have is from what is the state when
> you would upload this to packagemanager - and from my pov this would only
> alow you to unfold that package so you have all the contained packages
> available. And yes we might even add a ui / rest api feature to triggere
> installation for all of the contained packages automatically (e.g. the
> mutable subset or all if there is no CNS) yet the package itself would not
> have an installation state on its own as it is only a shipping container.
>
> Does this clarify my thoughts about this a bit better?
>
> When it comes to nesting of packages of same type it is ambiguous so we
> might want to clarify this for sure.
>
> Cheers
> Dominik
>
> On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <ko...@gmx.de> wrote:
>
>> Hi Dominik,
>> I am not getting exactly what you propose here:
>> Which types should be allowed in "container"?
>> a) If there is already a support for "content" type packages I don't see
>> the need for another type within "container"
>> b) If there is no support for nested content type packages, the question
>> is how to aggregate multiple content type packages together so that they
>> are actually installed via one big content-package?
>>
>> As I already said, I am not too much worried about a mixture of mutable
>> and immutable packages (that can still be done via 'mixed') but if we
>> support that in FileVault via a dedicated type, this should be installable
>> as well (i.e. be a regular content package).
>> Having a pure package format without supporting installation sounds like
>> a unnecessary limitation (even having AEM as a Cloud Service in mind)
>> because a lot of consumers right now don't care about mutable/immutable
>> (i.e. old AEM instances not leveraging the Composite Node Store).
>> As I already said, I would like to see a clarification ticket for
>> JCRVLT-170, because that clearly does neither mention nested containers nor
>> content type packages within containers (therefore everyone assumes, those
>> are forbidden)
>>
>> Thanks,
>> Konrad
>>
>> On 21. Jan 2020, at 08:34, Dominik Süß <do...@gmail.com> wrote:
>>
>> Hi Konrad,
>>
>> I get your point yet that was not really the origin. Probably we can
>> split this off in another way.
>> Let's introduce a PackageType.Collection - basically this would supposed
>> to be never installed but is just a shipping unit holding a set of other
>> packages. These could either be in a flat list or for compat reasons also
>> be  placed in an /etc/packages structure.
>> This packageType would be excluded of evaluation and is supposed to never
>> be installed directly. It basically is a zip file acting as a wrapper
>> around a collection of other packages.
>> No further semantics beside the fact that it has no vault identity on its
>> own and won't be visible after being installed - it's a pure shipping unit.
>>
>> Cheers
>> Dominik
>>
>> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <ko...@gmx.de> wrote:
>>
>>> Hi Dominik,
>>> For me container was always immutable by only embedding immutable
>>> content (bundles, config, app-packages, as indicated in JCRVLT-170)
>>> Especially when we allow nesting of containers, we should probably think
>>> of two different container types:
>>> - container (immutable), with the current restrictions, but in addition
>>> allows nested containers (immutable)
>>> - content-container (mutable), only allows content and content-container
>>> sub packages
>>>
>>> That way we have a clean separation even on the container level.
>>> Whether we need one container which allows embedding both container
>>> (immutable) and content-container (mutable), I am not sure about. WDYT?
>>> Also not very sure about the naming of such a construct...
>>>
>>> Konrad
>>>
>>> On 16. Jan 2020, at 16:40, Dominik Süß <do...@gmail.com> wrote:
>>>
>>> Hi Konrad,
>>>
>>> mixed is a legacy structure that basically allows all kinds of things.
>>> What we would want to avoid is mixing content and application content in
>>> the same package - building a container that allow application and content
>>> package to be grouped together (as installable entities) but making sure
>>> they are not mashed together helps to install both at distinct times. If
>>> you check setups working with the composite nodestore you would have 2
>>> different times at which the application and the content packages would be
>>> installed. Crucial is that containers basically don't really define an own
>>> identity that you would be depending on - you should only define
>>> dependencies on the actual installables . The problem of a mixed package is
>>> that people start to add content to it and therefore you have to define
>>> dependencies on it, if they then also contain application on content there
>>> is ambiguity when this package actually is in an installed state as the
>>> application part would need to be installed when preparing the immutable
>>> part of the repository while the content only can be installed at runtime
>>> with the mutable content being accessible.
>>> Containers would be the tool to declare a grouping of entities that are
>>> supposed to be installed while (optimally - not sure if we should enforce)
>>> not playing a role in the dependency handling at all.
>>>
>>> Cheers
>>> Dominik
>>>
>>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <ko...@gmx.de> wrote:
>>>
>>>> Hi Dominik,
>>>> better late feedback than no feedback at all.
>>>> Let me comment on your points
>>>>
>>>> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com> wrote:
>>>>
>>>> Based on the last changes in the filevault plugin some of the setups
>>>> that were in place are no longer working - this is specifically by
>>>> enforcing to not embed osgi bundles and configs.
>>>>
>>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>>
>>>>
>>>> My suggestion to make sure this is not breaking each and every existing
>>>> setup would be 2fold
>>>> a) make sure we can override those rules to run it in some kind of
>>>> legacy mode
>>>>
>>>> That is implicitly done by setting the packageType to 'mixed'.
>>>> But maybe I just don't understand correctly. I would help a lot to
>>>> discuss on a concrete example. Can you share one which is no longer working?
>>>>
>>>> b) we optimize the tooling to support a merging of container packages
>>>> into one container or provide a nesting capability
>>>>
>>>> There was a question just raised today on what a container is supposed
>>>> to support. According to
>>>> https://issues.apache.org/jira/browse/JCRVLT-170 neither nested
>>>> container packages nor content packages.
>>>> What exactly do you propose? Can you formulate your proposal in a
>>>> dedicated JIRA ticket for FileVault?
>>>>
>>>>
>>>> As an endresult it should be possible to slowly migrate existing
>>>> structures (that currently are postprocessed by a converter) towards
>>>> something that natively enables a suitable structure while the
>>>> codestructures would not need to change beyond the pom definitions.
>>>>
>>>> With 'mixed' this is already the case IMHO, other changes might need
>>>> further refactorings....
>>>>
>>>>
>>>> Right now in most cases because of compatiblity reasons with the osgi
>>>> installer the osgi configs and osgi bundles are part of the application
>>>> packages. This is failing for almost everyone - and if we want them to be
>>>> moved out into the container we must make sure that the structures can be
>>>> reflected.
>>>>
>>>> Not necessarily. Even the OSGi installer can deal with container
>>>> packages containing both app subpackages and bundles/configs.
>>>>
>>>>
>>>> As a first step I would suggest we establish some property to establish
>>>> some legacy mode for application packages and as a next step make sure we
>>>> get container packages that allow the same that is currently partially
>>>> reflected by application packages with mutliple install folders (e.g. with
>>>> runmode support).
>>>>
>>>> What exactly is wrong about 'mixed'?
>>>>
>>>>
>>>> As long as we want to support the sling installer for a while the
>>>> container structure itself would need be able to locate the binaries and
>>>> packages in locations the installer is aware of (install and config
>>>> folders) or we need to patch the sling installer to be able to process the
>>>> new container packages accurately
>>>>
>>>> Container packages allow exactly that IMHO
>>>>
>>>>
>>>> Cheers
>>>> Dominik
>>>>
>>>> Konrad
>>>>
>>>
>>>
>>
>

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

Hi Dominik,
This is still very vague to me. 

What is your idea about the current packageType: container (which is a content-package with an identity)? What should be allowed in there?
What about the new propsed type PackageType.Collection? How does that relate to the existing packageType? Does it replace it?

Can you try to answer those questions, so that everyone can get a clearer picture at what you are aiming?

I would strongly suggest not to come up with something which is not a content-package itself, because it seems no one has the time to implement new features in FileVault (like. a package which can only be unwrapped, but no one can depend on).
Let's rather enforce this with some validators: https://jackrabbit.apache.org/filevault/validation.html <https://jackrabbit.apache.org/filevault/validation.html>. Way less effort IMHO and much better backwards compatible!
Thanks,
Konrad


> On 21. Jan 2020, at 09:20, Dominik Süß <do...@gmail.com> wrote:
> 
> Hi Konrad,
> 
> what worries me about mixed is that is doesn't enforce any separation which is the scenario that we want to prevent. 
> A mixed content-package has an identity which another package can depend on - yet in a composite nodestore the mutable and immutable parts are installed at different times making it impossible to model the dependencies of immutable packages against immutable content tha would be served from a mixed package.
> This is why supporting mixed content packages cannot work in a composite nodestore scenario.
> 
> Yes it is possible to abuse mixed packages as what I propose as a collection type - yet no one prevents you from adding content directly to that package which in turn might require to set direct dependencies for it making it again something you cannot simply drop
> 
> Therefore the suggestion is to have a type that allows bundling together everything you want to eventually get installed into the system, where all packages that have an identity are clearly either application or content (so separation of mutable and immutable) so the dependency ordering is no longer ambiguous between deployment (change of immutable content state) and runtime (change of mutable state).
> 
> I understand that the perspective you have is from what is the state when you would upload this to packagemanager - and from my pov this would only alow you to unfold that package so you have all the contained packages available. And yes we might even add a ui / rest api feature to triggere installation for all of the contained packages automatically (e.g. the mutable subset or all if there is no CNS) yet the package itself would not have an installation state on its own as it is only a shipping container.
> 
> Does this clarify my thoughts about this a bit better?
> 
> When it comes to nesting of packages of same type it is ambiguous so we might want to clarify this for sure.
> 
> Cheers
> Dominik
> 
> On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
> Hi Dominik,
> I am not getting exactly what you propose here:
> Which types should be allowed in "container"?
> a) If there is already a support for "content" type packages I don't see the need for another type within "container"
> b) If there is no support for nested content type packages, the question is how to aggregate multiple content type packages together so that they are actually installed via one big content-package?
> 
> As I already said, I am not too much worried about a mixture of mutable and immutable packages (that can still be done via 'mixed') but if we support that in FileVault via a dedicated type, this should be installable as well (i.e. be a regular content package).
> Having a pure package format without supporting installation sounds like a unnecessary limitation (even having AEM as a Cloud Service in mind) because a lot of consumers right now don't care about mutable/immutable (i.e. old AEM instances not leveraging the Composite Node Store).
> As I already said, I would like to see a clarification ticket for JCRVLT-170, because that clearly does neither mention nested containers nor content type packages within containers (therefore everyone assumes, those are forbidden)
> 
> Thanks,
> Konrad
> 
>> On 21. Jan 2020, at 08:34, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi Konrad,
>> 
>> I get your point yet that was not really the origin. Probably we can split this off in another way.
>> Let's introduce a PackageType.Collection - basically this would supposed to be never installed but is just a shipping unit holding a set of other packages. These could either be in a flat list or for compat reasons also be  placed in an /etc/packages structure.
>> This packageType would be excluded of evaluation and is supposed to never be installed directly. It basically is a zip file acting as a wrapper around a collection of other packages.
>> No further semantics beside the fact that it has no vault identity on its own and won't be visible after being installed - it's a pure shipping unit.
>> 
>> Cheers
>> Dominik
>> 
>> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>> Hi Dominik,
>> For me container was always immutable by only embedding immutable content (bundles, config, app-packages, as indicated in JCRVLT-170)
>> Especially when we allow nesting of containers, we should probably think of two different container types:
>> - container (immutable), with the current restrictions, but in addition allows nested containers (immutable)
>> - content-container (mutable), only allows content and content-container sub packages
>> 
>> That way we have a clean separation even on the container level.
>> Whether we need one container which allows embedding both container (immutable) and content-container (mutable), I am not sure about. WDYT? Also not very sure about the naming of such a construct...
>> 
>> Konrad
>> 
>>> On 16. Jan 2020, at 16:40, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Hi Konrad,
>>> 
>>> mixed is a legacy structure that basically allows all kinds of things. What we would want to avoid is mixing content and application content in the same package - building a container that allow application and content package to be grouped together (as installable entities) but making sure they are not mashed together helps to install both at distinct times. If you check setups working with the composite nodestore you would have 2 different times at which the application and the content packages would be installed. Crucial is that containers basically don't really define an own identity that you would be depending on - you should only define dependencies on the actual installables . The problem of a mixed package is that people start to add content to it and therefore you have to define dependencies on it, if they then also contain application on content there is ambiguity when this package actually is in an installed state as the application part would need to be installed when preparing the immutable part of the repository while the content only can be installed at runtime with the mutable content being accessible.
>>> Containers would be the tool to declare a grouping of entities that are supposed to be installed while (optimally - not sure if we should enforce) not playing a role in the dependency handling at all.
>>> 
>>> Cheers
>>> Dominik
>>> 
>>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>>> Hi Dominik,
>>> better late feedback than no feedback at all.
>>> Let me comment on your points
>>> 
>>>> On 16. Jan 2020, at 15:09, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>>> 
>>>> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>> 
>>>> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
>>>> a) make sure we can override those rules to run it in some kind of legacy mode
>>> That is implicitly done by setting the packageType to 'mixed'.
>>> But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?
>>> 
>>>> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
>>> There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
>>> What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
>>>> 
>>>> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
>>> With 'mixed' this is already the case IMHO, other changes might need further refactorings....
>>>> 
>>>> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
>>> Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
>>>> 
>>>> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
>>> What exactly is wrong about 'mixed'?
>>>> 
>>>> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
>>> Container packages allow exactly that IMHO
>>>> 
>>>> Cheers
>>>> Dominik
>>>> 
>>> Konrad
>> 
> 

Re: Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

Hi Konrad,

what worries me about mixed is that is doesn't enforce any separation which
is the scenario that we want to prevent.
A mixed content-package has an identity which another package can depend on
- yet in a composite nodestore the mutable and immutable parts are
installed at different times making it impossible to model the dependencies
of immutable packages against immutable content tha would be served from a
mixed package.
This is why supporting mixed content packages cannot work in a composite
nodestore scenario.

Yes it is possible to abuse mixed packages as what I propose as a
collection type - yet no one prevents you from adding content directly to
that package which in turn might require to set direct dependencies for it
making it again something you cannot simply drop

Therefore the suggestion is to have a type that allows bundling together
everything you want to eventually get installed into the system, where all
packages that have an identity are clearly either application or content
(so separation of mutable and immutable) so the dependency ordering is no
longer ambiguous between deployment (change of immutable content state) and
runtime (change of mutable state).

I understand that the perspective you have is from what is the state when
you would upload this to packagemanager - and from my pov this would only
alow you to unfold that package so you have all the contained packages
available. And yes we might even add a ui / rest api feature to triggere
installation for all of the contained packages automatically (e.g. the
mutable subset or all if there is no CNS) yet the package itself would not
have an installation state on its own as it is only a shipping container.

Does this clarify my thoughts about this a bit better?

When it comes to nesting of packages of same type it is ambiguous so we
might want to clarify this for sure.

Cheers
Dominik

On Tue, Jan 21, 2020 at 8:47 AM Konrad Windszus <ko...@gmx.de> wrote:

> Hi Dominik,
> I am not getting exactly what you propose here:
> Which types should be allowed in "container"?
> a) If there is already a support for "content" type packages I don't see
> the need for another type within "container"
> b) If there is no support for nested content type packages, the question
> is how to aggregate multiple content type packages together so that they
> are actually installed via one big content-package?
>
> As I already said, I am not too much worried about a mixture of mutable
> and immutable packages (that can still be done via 'mixed') but if we
> support that in FileVault via a dedicated type, this should be installable
> as well (i.e. be a regular content package).
> Having a pure package format without supporting installation sounds like a
> unnecessary limitation (even having AEM as a Cloud Service in mind) because
> a lot of consumers right now don't care about mutable/immutable (i.e. old
> AEM instances not leveraging the Composite Node Store).
> As I already said, I would like to see a clarification ticket for
> JCRVLT-170, because that clearly does neither mention nested containers nor
> content type packages within containers (therefore everyone assumes, those
> are forbidden)
>
> Thanks,
> Konrad
>
> On 21. Jan 2020, at 08:34, Dominik Süß <do...@gmail.com> wrote:
>
> Hi Konrad,
>
> I get your point yet that was not really the origin. Probably we can split
> this off in another way.
> Let's introduce a PackageType.Collection - basically this would supposed
> to be never installed but is just a shipping unit holding a set of other
> packages. These could either be in a flat list or for compat reasons also
> be  placed in an /etc/packages structure.
> This packageType would be excluded of evaluation and is supposed to never
> be installed directly. It basically is a zip file acting as a wrapper
> around a collection of other packages.
> No further semantics beside the fact that it has no vault identity on its
> own and won't be visible after being installed - it's a pure shipping unit.
>
> Cheers
> Dominik
>
> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <ko...@gmx.de> wrote:
>
>> Hi Dominik,
>> For me container was always immutable by only embedding immutable content
>> (bundles, config, app-packages, as indicated in JCRVLT-170)
>> Especially when we allow nesting of containers, we should probably think
>> of two different container types:
>> - container (immutable), with the current restrictions, but in addition
>> allows nested containers (immutable)
>> - content-container (mutable), only allows content and content-container
>> sub packages
>>
>> That way we have a clean separation even on the container level.
>> Whether we need one container which allows embedding both container
>> (immutable) and content-container (mutable), I am not sure about. WDYT?
>> Also not very sure about the naming of such a construct...
>>
>> Konrad
>>
>> On 16. Jan 2020, at 16:40, Dominik Süß <do...@gmail.com> wrote:
>>
>> Hi Konrad,
>>
>> mixed is a legacy structure that basically allows all kinds of things.
>> What we would want to avoid is mixing content and application content in
>> the same package - building a container that allow application and content
>> package to be grouped together (as installable entities) but making sure
>> they are not mashed together helps to install both at distinct times. If
>> you check setups working with the composite nodestore you would have 2
>> different times at which the application and the content packages would be
>> installed. Crucial is that containers basically don't really define an own
>> identity that you would be depending on - you should only define
>> dependencies on the actual installables . The problem of a mixed package is
>> that people start to add content to it and therefore you have to define
>> dependencies on it, if they then also contain application on content there
>> is ambiguity when this package actually is in an installed state as the
>> application part would need to be installed when preparing the immutable
>> part of the repository while the content only can be installed at runtime
>> with the mutable content being accessible.
>> Containers would be the tool to declare a grouping of entities that are
>> supposed to be installed while (optimally - not sure if we should enforce)
>> not playing a role in the dependency handling at all.
>>
>> Cheers
>> Dominik
>>
>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <ko...@gmx.de> wrote:
>>
>>> Hi Dominik,
>>> better late feedback than no feedback at all.
>>> Let me comment on your points
>>>
>>> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com> wrote:
>>>
>>> Based on the last changes in the filevault plugin some of the setups
>>> that were in place are no longer working - this is specifically by
>>> enforcing to not embed osgi bundles and configs.
>>>
>>> IMHO not if one explicitly sets the packageType to 'mixed'
>>>
>>>
>>> My suggestion to make sure this is not breaking each and every existing
>>> setup would be 2fold
>>> a) make sure we can override those rules to run it in some kind of
>>> legacy mode
>>>
>>> That is implicitly done by setting the packageType to 'mixed'.
>>> But maybe I just don't understand correctly. I would help a lot to
>>> discuss on a concrete example. Can you share one which is no longer working?
>>>
>>> b) we optimize the tooling to support a merging of container packages
>>> into one container or provide a nesting capability
>>>
>>> There was a question just raised today on what a container is supposed
>>> to support. According to
>>> https://issues.apache.org/jira/browse/JCRVLT-170 neither nested
>>> container packages nor content packages.
>>> What exactly do you propose? Can you formulate your proposal in a
>>> dedicated JIRA ticket for FileVault?
>>>
>>>
>>> As an endresult it should be possible to slowly migrate existing
>>> structures (that currently are postprocessed by a converter) towards
>>> something that natively enables a suitable structure while the
>>> codestructures would not need to change beyond the pom definitions.
>>>
>>> With 'mixed' this is already the case IMHO, other changes might need
>>> further refactorings....
>>>
>>>
>>> Right now in most cases because of compatiblity reasons with the osgi
>>> installer the osgi configs and osgi bundles are part of the application
>>> packages. This is failing for almost everyone - and if we want them to be
>>> moved out into the container we must make sure that the structures can be
>>> reflected.
>>>
>>> Not necessarily. Even the OSGi installer can deal with container
>>> packages containing both app subpackages and bundles/configs.
>>>
>>>
>>> As a first step I would suggest we establish some property to establish
>>> some legacy mode for application packages and as a next step make sure we
>>> get container packages that allow the same that is currently partially
>>> reflected by application packages with mutliple install folders (e.g. with
>>> runmode support).
>>>
>>> What exactly is wrong about 'mixed'?
>>>
>>>
>>> As long as we want to support the sling installer for a while the
>>> container structure itself would need be able to locate the binaries and
>>> packages in locations the installer is aware of (install and config
>>> folders) or we need to patch the sling installer to be able to process the
>>> new container packages accurately
>>>
>>> Container packages allow exactly that IMHO
>>>
>>>
>>> Cheers
>>> Dominik
>>>
>>> Konrad
>>>
>>
>>
>

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

Hi Dominik,
I am not getting exactly what you propose here:
Which types should be allowed in "container"?
a) If there is already a support for "content" type packages I don't see the need for another type within "container"
b) If there is no support for nested content type packages, the question is how to aggregate multiple content type packages together so that they are actually installed via one big content-package?

As I already said, I am not too much worried about a mixture of mutable and immutable packages (that can still be done via 'mixed') but if we support that in FileVault via a dedicated type, this should be installable as well (i.e. be a regular content package).
Having a pure package format without supporting installation sounds like a unnecessary limitation (even having AEM as a Cloud Service in mind) because a lot of consumers right now don't care about mutable/immutable (i.e. old AEM instances not leveraging the Composite Node Store).
As I already said, I would like to see a clarification ticket for JCRVLT-170, because that clearly does neither mention nested containers nor content type packages within containers (therefore everyone assumes, those are forbidden)

Thanks,
Konrad

> On 21. Jan 2020, at 08:34, Dominik Süß <do...@gmail.com> wrote:
> 
> Hi Konrad,
> 
> I get your point yet that was not really the origin. Probably we can split this off in another way.
> Let's introduce a PackageType.Collection - basically this would supposed to be never installed but is just a shipping unit holding a set of other packages. These could either be in a flat list or for compat reasons also be  placed in an /etc/packages structure.
> This packageType would be excluded of evaluation and is supposed to never be installed directly. It basically is a zip file acting as a wrapper around a collection of other packages.
> No further semantics beside the fact that it has no vault identity on its own and won't be visible after being installed - it's a pure shipping unit.
> 
> Cheers
> Dominik
> 
> On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
> Hi Dominik,
> For me container was always immutable by only embedding immutable content (bundles, config, app-packages, as indicated in JCRVLT-170)
> Especially when we allow nesting of containers, we should probably think of two different container types:
> - container (immutable), with the current restrictions, but in addition allows nested containers (immutable)
> - content-container (mutable), only allows content and content-container sub packages
> 
> That way we have a clean separation even on the container level.
> Whether we need one container which allows embedding both container (immutable) and content-container (mutable), I am not sure about. WDYT? Also not very sure about the naming of such a construct...
> 
> Konrad
> 
>> On 16. Jan 2020, at 16:40, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi Konrad,
>> 
>> mixed is a legacy structure that basically allows all kinds of things. What we would want to avoid is mixing content and application content in the same package - building a container that allow application and content package to be grouped together (as installable entities) but making sure they are not mashed together helps to install both at distinct times. If you check setups working with the composite nodestore you would have 2 different times at which the application and the content packages would be installed. Crucial is that containers basically don't really define an own identity that you would be depending on - you should only define dependencies on the actual installables . The problem of a mixed package is that people start to add content to it and therefore you have to define dependencies on it, if they then also contain application on content there is ambiguity when this package actually is in an installed state as the application part would need to be installed when preparing the immutable part of the repository while the content only can be installed at runtime with the mutable content being accessible.
>> Containers would be the tool to declare a grouping of entities that are supposed to be installed while (optimally - not sure if we should enforce) not playing a role in the dependency handling at all.
>> 
>> Cheers
>> Dominik
>> 
>> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
>> Hi Dominik,
>> better late feedback than no feedback at all.
>> Let me comment on your points
>> 
>>> On 16. Jan 2020, at 15:09, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
>> IMHO not if one explicitly sets the packageType to 'mixed'
>>> 
>>> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
>>> a) make sure we can override those rules to run it in some kind of legacy mode
>> That is implicitly done by setting the packageType to 'mixed'.
>> But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?
>> 
>>> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
>> There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
>> What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
>>> 
>>> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
>> With 'mixed' this is already the case IMHO, other changes might need further refactorings....
>>> 
>>> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
>> Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
>>> 
>>> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
>> What exactly is wrong about 'mixed'?
>>> 
>>> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
>> Container packages allow exactly that IMHO
>>> 
>>> Cheers
>>> Dominik
>>> 
>> Konrad
> 

Re: Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

Hi Konrad,

I get your point yet that was not really the origin. Probably we can split
this off in another way.
Let's introduce a PackageType.Collection - basically this would supposed to
be never installed but is just a shipping unit holding a set of other
packages. These could either be in a flat list or for compat reasons also
be  placed in an /etc/packages structure.
This packageType would be excluded of evaluation and is supposed to never
be installed directly. It basically is a zip file acting as a wrapper
around a collection of other packages.
No further semantics beside the fact that it has no vault identity on its
own and won't be visible after being installed - it's a pure shipping unit.

Cheers
Dominik

On Thu, Jan 16, 2020 at 5:21 PM Konrad Windszus <ko...@gmx.de> wrote:

> Hi Dominik,
> For me container was always immutable by only embedding immutable content
> (bundles, config, app-packages, as indicated in JCRVLT-170)
> Especially when we allow nesting of containers, we should probably think
> of two different container types:
> - container (immutable), with the current restrictions, but in addition
> allows nested containers (immutable)
> - content-container (mutable), only allows content and content-container
> sub packages
>
> That way we have a clean separation even on the container level.
> Whether we need one container which allows embedding both container
> (immutable) and content-container (mutable), I am not sure about. WDYT?
> Also not very sure about the naming of such a construct...
>
> Konrad
>
> On 16. Jan 2020, at 16:40, Dominik Süß <do...@gmail.com> wrote:
>
> Hi Konrad,
>
> mixed is a legacy structure that basically allows all kinds of things.
> What we would want to avoid is mixing content and application content in
> the same package - building a container that allow application and content
> package to be grouped together (as installable entities) but making sure
> they are not mashed together helps to install both at distinct times. If
> you check setups working with the composite nodestore you would have 2
> different times at which the application and the content packages would be
> installed. Crucial is that containers basically don't really define an own
> identity that you would be depending on - you should only define
> dependencies on the actual installables . The problem of a mixed package is
> that people start to add content to it and therefore you have to define
> dependencies on it, if they then also contain application on content there
> is ambiguity when this package actually is in an installed state as the
> application part would need to be installed when preparing the immutable
> part of the repository while the content only can be installed at runtime
> with the mutable content being accessible.
> Containers would be the tool to declare a grouping of entities that are
> supposed to be installed while (optimally - not sure if we should enforce)
> not playing a role in the dependency handling at all.
>
> Cheers
> Dominik
>
> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <ko...@gmx.de> wrote:
>
>> Hi Dominik,
>> better late feedback than no feedback at all.
>> Let me comment on your points
>>
>> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com> wrote:
>>
>> Based on the last changes in the filevault plugin some of the setups that
>> were in place are no longer working - this is specifically by enforcing to
>> not embed osgi bundles and configs.
>>
>> IMHO not if one explicitly sets the packageType to 'mixed'
>>
>>
>> My suggestion to make sure this is not breaking each and every existing
>> setup would be 2fold
>> a) make sure we can override those rules to run it in some kind of legacy
>> mode
>>
>> That is implicitly done by setting the packageType to 'mixed'.
>> But maybe I just don't understand correctly. I would help a lot to
>> discuss on a concrete example. Can you share one which is no longer working?
>>
>> b) we optimize the tooling to support a merging of container packages
>> into one container or provide a nesting capability
>>
>> There was a question just raised today on what a container is supposed to
>> support. According to https://issues.apache.org/jira/browse/JCRVLT-170 neither
>> nested container packages nor content packages.
>> What exactly do you propose? Can you formulate your proposal in a
>> dedicated JIRA ticket for FileVault?
>>
>>
>> As an endresult it should be possible to slowly migrate existing
>> structures (that currently are postprocessed by a converter) towards
>> something that natively enables a suitable structure while the
>> codestructures would not need to change beyond the pom definitions.
>>
>> With 'mixed' this is already the case IMHO, other changes might need
>> further refactorings....
>>
>>
>> Right now in most cases because of compatiblity reasons with the osgi
>> installer the osgi configs and osgi bundles are part of the application
>> packages. This is failing for almost everyone - and if we want them to be
>> moved out into the container we must make sure that the structures can be
>> reflected.
>>
>> Not necessarily. Even the OSGi installer can deal with container packages
>> containing both app subpackages and bundles/configs.
>>
>>
>> As a first step I would suggest we establish some property to establish
>> some legacy mode for application packages and as a next step make sure we
>> get container packages that allow the same that is currently partially
>> reflected by application packages with mutliple install folders (e.g. with
>> runmode support).
>>
>> What exactly is wrong about 'mixed'?
>>
>>
>> As long as we want to support the sling installer for a while the
>> container structure itself would need be able to locate the binaries and
>> packages in locations the installer is aware of (install and config
>> folders) or we need to patch the sling installer to be able to process the
>> new container packages accurately
>>
>> Container packages allow exactly that IMHO
>>
>>
>> Cheers
>> Dominik
>>
>> Konrad
>>
>
>

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

Hi Dominik,
For me container was always immutable by only embedding immutable content (bundles, config, app-packages, as indicated in JCRVLT-170)
Especially when we allow nesting of containers, we should probably think of two different container types:
- container (immutable), with the current restrictions, but in addition allows nested containers (immutable)
- content-container (mutable), only allows content and content-container sub packages

That way we have a clean separation even on the container level.
Whether we need one container which allows embedding both container (immutable) and content-container (mutable), I am not sure about. WDYT? Also not very sure about the naming of such a construct...

Konrad

> On 16. Jan 2020, at 16:40, Dominik Süß <do...@gmail.com> wrote:
> 
> Hi Konrad,
> 
> mixed is a legacy structure that basically allows all kinds of things. What we would want to avoid is mixing content and application content in the same package - building a container that allow application and content package to be grouped together (as installable entities) but making sure they are not mashed together helps to install both at distinct times. If you check setups working with the composite nodestore you would have 2 different times at which the application and the content packages would be installed. Crucial is that containers basically don't really define an own identity that you would be depending on - you should only define dependencies on the actual installables . The problem of a mixed package is that people start to add content to it and therefore you have to define dependencies on it, if they then also contain application on content there is ambiguity when this package actually is in an installed state as the application part would need to be installed when preparing the immutable part of the repository while the content only can be installed at runtime with the mutable content being accessible.
> Containers would be the tool to declare a grouping of entities that are supposed to be installed while (optimally - not sure if we should enforce) not playing a role in the dependency handling at all.
> 
> Cheers
> Dominik
> 
> On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <konrad_w@gmx.de <ma...@gmx.de>> wrote:
> Hi Dominik,
> better late feedback than no feedback at all.
> Let me comment on your points
> 
>> On 16. Jan 2020, at 15:09, Dominik Süß <dominik.suess@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
> IMHO not if one explicitly sets the packageType to 'mixed'
>> 
>> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
>> a) make sure we can override those rules to run it in some kind of legacy mode
> That is implicitly done by setting the packageType to 'mixed'.
> But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?
> 
>> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
> There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
> What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
>> 
>> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
> With 'mixed' this is already the case IMHO, other changes might need further refactorings....
>> 
>> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
> Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
>> 
>> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
> What exactly is wrong about 'mixed'?
>> 
>> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
> Container packages allow exactly that IMHO
>> 
>> Cheers
>> Dominik
>> 
> Konrad

Re: Impact of maven filevault plugin validation on existing setups

[Dominik Süß <do...@gmail.com>]

Hi Konrad,

mixed is a legacy structure that basically allows all kinds of things. What
we would want to avoid is mixing content and application content in the
same package - building a container that allow application and content
package to be grouped together (as installable entities) but making sure
they are not mashed together helps to install both at distinct times. If
you check setups working with the composite nodestore you would have 2
different times at which the application and the content packages would be
installed. Crucial is that containers basically don't really define an own
identity that you would be depending on - you should only define
dependencies on the actual installables . The problem of a mixed package is
that people start to add content to it and therefore you have to define
dependencies on it, if they then also contain application on content there
is ambiguity when this package actually is in an installed state as the
application part would need to be installed when preparing the immutable
part of the repository while the content only can be installed at runtime
with the mutable content being accessible.
Containers would be the tool to declare a grouping of entities that are
supposed to be installed while (optimally - not sure if we should enforce)
not playing a role in the dependency handling at all.

Cheers
Dominik

On Thu, Jan 16, 2020 at 3:33 PM Konrad Windszus <ko...@gmx.de> wrote:

> Hi Dominik,
> better late feedback than no feedback at all.
> Let me comment on your points
>
> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com> wrote:
>
> Based on the last changes in the filevault plugin some of the setups that
> were in place are no longer working - this is specifically by enforcing to
> not embed osgi bundles and configs.
>
> IMHO not if one explicitly sets the packageType to 'mixed'
>
>
> My suggestion to make sure this is not breaking each and every existing
> setup would be 2fold
> a) make sure we can override those rules to run it in some kind of legacy
> mode
>
> That is implicitly done by setting the packageType to 'mixed'.
> But maybe I just don't understand correctly. I would help a lot to discuss
> on a concrete example. Can you share one which is no longer working?
>
> b) we optimize the tooling to support a merging of container packages into
> one container or provide a nesting capability
>
> There was a question just raised today on what a container is supposed to
> support. According to https://issues.apache.org/jira/browse/JCRVLT-170 neither
> nested container packages nor content packages.
> What exactly do you propose? Can you formulate your proposal in a
> dedicated JIRA ticket for FileVault?
>
>
> As an endresult it should be possible to slowly migrate existing
> structures (that currently are postprocessed by a converter) towards
> something that natively enables a suitable structure while the
> codestructures would not need to change beyond the pom definitions.
>
> With 'mixed' this is already the case IMHO, other changes might need
> further refactorings....
>
>
> Right now in most cases because of compatiblity reasons with the osgi
> installer the osgi configs and osgi bundles are part of the application
> packages. This is failing for almost everyone - and if we want them to be
> moved out into the container we must make sure that the structures can be
> reflected.
>
> Not necessarily. Even the OSGi installer can deal with container packages
> containing both app subpackages and bundles/configs.
>
>
> As a first step I would suggest we establish some property to establish
> some legacy mode for application packages and as a next step make sure we
> get container packages that allow the same that is currently partially
> reflected by application packages with mutliple install folders (e.g. with
> runmode support).
>
> What exactly is wrong about 'mixed'?
>
>
> As long as we want to support the sling installer for a while the
> container structure itself would need be able to locate the binaries and
> packages in locations the installer is aware of (install and config
> folders) or we need to patch the sling installer to be able to process the
> new container packages accurately
>
> Container packages allow exactly that IMHO
>
>
> Cheers
> Dominik
>
> Konrad
>

Re: Impact of maven filevault plugin validation on existing setups

[Konrad Windszus <ko...@gmx.de>]

Hi Dominik,
better late feedback than no feedback at all.
Let me comment on your points

> On 16. Jan 2020, at 15:09, Dominik Süß <do...@gmail.com> wrote:
> 
> Based on the last changes in the filevault plugin some of the setups that were in place are no longer working - this is specifically by enforcing to not embed osgi bundles and configs.
IMHO not if one explicitly sets the packageType to 'mixed'
> 
> My suggestion to make sure this is not breaking each and every existing setup would be 2fold
> a) make sure we can override those rules to run it in some kind of legacy mode
That is implicitly done by setting the packageType to 'mixed'.
But maybe I just don't understand correctly. I would help a lot to discuss on a concrete example. Can you share one which is no longer working?

> b) we optimize the tooling to support a merging of container packages into one container or provide a nesting capability
There was a question just raised today on what a container is supposed to support. According to https://issues.apache.org/jira/browse/JCRVLT-170 <https://issues.apache.org/jira/browse/JCRVLT-170> neither nested container packages nor content packages.
What exactly do you propose? Can you formulate your proposal in a dedicated JIRA ticket for FileVault?
> 
> As an endresult it should be possible to slowly migrate existing structures (that currently are postprocessed by a converter) towards something that natively enables a suitable structure while the codestructures would not need to change beyond the pom definitions. 
With 'mixed' this is already the case IMHO, other changes might need further refactorings....
> 
> Right now in most cases because of compatiblity reasons with the osgi installer the osgi configs and osgi bundles are part of the application packages. This is failing for almost everyone - and if we want them to be moved out into the container we must make sure that the structures can be reflected.
Not necessarily. Even the OSGi installer can deal with container packages containing both app subpackages and bundles/configs.
> 
> As a first step I would suggest we establish some property to establish some legacy mode for application packages and as a next step make sure we get container packages that allow the same that is currently partially reflected by application packages with mutliple install folders (e.g. with runmode support). 
What exactly is wrong about 'mixed'?
> 
> As long as we want to support the sling installer for a while the container structure itself would need be able to locate the binaries and packages in locations the installer is aware of (install and config folders) or we need to patch the sling installer to be able to process the new container packages accurately 
Container packages allow exactly that IMHO
> 
> Cheers
> Dominik
> 
Konrad