[Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

Hi,

with the older provisioning model we support an archive containing a 
provisioning model and all contained artifacts ([1] and [2])
In addition we have a plugin for the OSGi installer which allows to 
install such artifacts ([3]).

The question is if we want to support something similar with the feature 
model?

Adjusting the code to support feature model based archives is very 
trivial, question is just if we want to do it and how ?

Regards
Carsten


[1] 
https://github.com/apache/sling-org-apache-sling-provisioning-model/blob/master/src/main/java/org/apache/sling/provisioning/model/io/ModelArchiveReader.java
[2] 
https://github.com/apache/sling-org-apache-sling-provisioning-model/blob/master/src/main/java/org/apache/sling/provisioning/model/io/ModelArchiveWriter.java
[3] https://github.com/apache/sling-org-apache-sling-installer-factory-model
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[David Bosschaert <da...@gmail.com>]

Hi all,

I think having a feature model archive is a great idea.

Besides the binary archives, the feature model archive also contains the
feature model itself. This can become large-ish, especially if extensions
are used to hold additional metadata. A couple of 100 Kb is possible. These
files will benefit from compression, although depending on your setup such
files might be very small compared to the binaries...

Best regards,

David

On Wed, 19 Feb 2020 at 06:12, Carsten Ziegeler <cz...@apache.org> wrote:

> Having an option for compressing sounds good to me, but I get a little
> bit worried to support two extensions. It is transparent for the user of
> the archive. As the archive usually contains binaries (which again might
> be compressed already), I'm not sure if there is a significant benefit
> in compressing the archive (in terms of the size of the archive).
>
> Regards
> Carsten
>
> On 18.02.2020 11:23, Bertrand Delacretaz wrote:
> > On Tue, Feb 18, 2020 at 11:00 AM Karl Pauls <ka...@gmail.com> wrote:
> >> ...could we maybe make
> >> it an option to create that zip file uncompressed?...
> >
> > If you need this I would suggest using different extensions for the
> > compressed and uncompressed formats - I suspect builds would add both
> > to the Maven repository in this case?
> >
> > So maybe .far and .zfar
> >
> > -Bertrand (refraining hard form suggesting .near and .far)
> >
>
> --
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org
>

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

Interesting, I came actually to the same conclusion already implemented 
it that way 😉

Carsten

On 19.02.2020 17:10, Julian Sedding wrote:
> Hi all
> 
> The ZIP file format (and the Java API) allows different compression levels
> per entry. Couldn’t we just not compress all embedded binary artifacts and
> have the best of both worlds? That is, assuming we know which files need to
> be uncompressed for the use case Karl mentioned.
> 
> Regards
> Julian
> 
> 
> 
> 
> 
> 
> On Wed, 19 Feb 2020 at 11:11, Carsten Ziegeler <cz...@apache.org> wrote:
> 
>> We could also simplify and say the archive is never compressed and we
>> only support that. I don't think that not compressing is a problem for
>> the binaries we put in there. If you have a huge feature model, then it
>> might make a difference as David points out. But I would also argue that
>> in that case you have a lot of artifacts. So again, it would not make
>> that much of a difference
>>
>> In addition - and that's probably not clear from my proposal - you can
>> use any extension with these archives. However, if we think of
>> supporting them in some tools, for example the OSGi installer, then we
>> need to detect such archives. Using an extension for that sounds like a
>> natural fit; however, we could also simply look at the manifest of the
>> archive. That's for example what we do for content packages which
>> usually have the general extension zip.
>>
>> So in other words, we don't need to worry about defining an extension
>> and just look at the manifest of the provided zip file.
>>
>> And we could either make compression configurable or not compress at all.
>>
>> WDYT?
>>
>> Carsten
>>
>> On 19.02.2020 11:06, Karl Pauls wrote:
>>> I'm not sure we need to have a second extension. I would assume the
>>> archive can be attached via a classifier anyways - hence, if you want
>>> to provide both, an uncompressed and a compressed version, I guess you
>>> can just encode the difference in the classifier name. Personally, I
>>> would be fine with just making the extension be "zip" to begin with
>>> since it sounds like that is all it is anyways (at least for now) but
>>> I don't really feel strongly about it - if I can use it to create
>>> uncompressed zips with everything in it I'm a happy camper already
>>> :-).
>>>
>>> regards,
>>>
>>> Karl
>>>
>>> On Wed, Feb 19, 2020 at 11:00 AM Bertrand Delacretaz
>>> <bd...@apache.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Wed, Feb 19, 2020 at 7:12 AM Carsten Ziegeler <cz...@apache.org>
>> wrote:
>>>>> ...Having an option for compressing sounds good to me, but I get a
>> little
>>>>> bit worried to support two extensions. It is transparent for the user
>> of
>>>>> the archive..
>>>>
>>>> Karl seems to imply that having an uncompressed archive is required
>>>> for some use cases so I suppose there's a need for the "consumers" of
>>>> those archives to be able to specify that they require the
>>>> uncompressed version?
>>>>
>>>> If that's correct, a way of identifying those uncompressed versions is
>>>> required, IMHO.
>>>>
>>>> That can be the file extension, or maybe a Maven classifier?
>>>>
>>>> A common pattern used for tar archives is to add a suffix to indicate
>>>> compression, such as .tar.gz
>>>>
>>>> -Bertrand
>>>
>>>
>>>
>>
>> --
>> --
>> Carsten Ziegeler
>> Adobe Research Switzerland
>> cziegeler@apache.org
>>
> 

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[Julian Sedding <js...@apache.org>]

Hi all

The ZIP file format (and the Java API) allows different compression levels
per entry. Couldn’t we just not compress all embedded binary artifacts and
have the best of both worlds? That is, assuming we know which files need to
be uncompressed for the use case Karl mentioned.

Regards
Julian






On Wed, 19 Feb 2020 at 11:11, Carsten Ziegeler <cz...@apache.org> wrote:

> We could also simplify and say the archive is never compressed and we
> only support that. I don't think that not compressing is a problem for
> the binaries we put in there. If you have a huge feature model, then it
> might make a difference as David points out. But I would also argue that
> in that case you have a lot of artifacts. So again, it would not make
> that much of a difference
>
> In addition - and that's probably not clear from my proposal - you can
> use any extension with these archives. However, if we think of
> supporting them in some tools, for example the OSGi installer, then we
> need to detect such archives. Using an extension for that sounds like a
> natural fit; however, we could also simply look at the manifest of the
> archive. That's for example what we do for content packages which
> usually have the general extension zip.
>
> So in other words, we don't need to worry about defining an extension
> and just look at the manifest of the provided zip file.
>
> And we could either make compression configurable or not compress at all.
>
> WDYT?
>
> Carsten
>
> On 19.02.2020 11:06, Karl Pauls wrote:
> > I'm not sure we need to have a second extension. I would assume the
> > archive can be attached via a classifier anyways - hence, if you want
> > to provide both, an uncompressed and a compressed version, I guess you
> > can just encode the difference in the classifier name. Personally, I
> > would be fine with just making the extension be "zip" to begin with
> > since it sounds like that is all it is anyways (at least for now) but
> > I don't really feel strongly about it - if I can use it to create
> > uncompressed zips with everything in it I'm a happy camper already
> > :-).
> >
> > regards,
> >
> > Karl
> >
> > On Wed, Feb 19, 2020 at 11:00 AM Bertrand Delacretaz
> > <bd...@apache.org> wrote:
> >>
> >> Hi,
> >>
> >> On Wed, Feb 19, 2020 at 7:12 AM Carsten Ziegeler <cz...@apache.org>
> wrote:
> >>> ...Having an option for compressing sounds good to me, but I get a
> little
> >>> bit worried to support two extensions. It is transparent for the user
> of
> >>> the archive..
> >>
> >> Karl seems to imply that having an uncompressed archive is required
> >> for some use cases so I suppose there's a need for the "consumers" of
> >> those archives to be able to specify that they require the
> >> uncompressed version?
> >>
> >> If that's correct, a way of identifying those uncompressed versions is
> >> required, IMHO.
> >>
> >> That can be the file extension, or maybe a Maven classifier?
> >>
> >> A common pattern used for tar archives is to add a suffix to indicate
> >> compression, such as .tar.gz
> >>
> >> -Bertrand
> >
> >
> >
>
> --
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org
>

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

We could also simplify and say the archive is never compressed and we 
only support that. I don't think that not compressing is a problem for 
the binaries we put in there. If you have a huge feature model, then it 
might make a difference as David points out. But I would also argue that 
in that case you have a lot of artifacts. So again, it would not make 
that much of a difference

In addition - and that's probably not clear from my proposal - you can 
use any extension with these archives. However, if we think of 
supporting them in some tools, for example the OSGi installer, then we 
need to detect such archives. Using an extension for that sounds like a 
natural fit; however, we could also simply look at the manifest of the 
archive. That's for example what we do for content packages which 
usually have the general extension zip.

So in other words, we don't need to worry about defining an extension 
and just look at the manifest of the provided zip file.

And we could either make compression configurable or not compress at all.

WDYT?

Carsten

On 19.02.2020 11:06, Karl Pauls wrote:
> I'm not sure we need to have a second extension. I would assume the
> archive can be attached via a classifier anyways - hence, if you want
> to provide both, an uncompressed and a compressed version, I guess you
> can just encode the difference in the classifier name. Personally, I
> would be fine with just making the extension be "zip" to begin with
> since it sounds like that is all it is anyways (at least for now) but
> I don't really feel strongly about it - if I can use it to create
> uncompressed zips with everything in it I'm a happy camper already
> :-).
> 
> regards,
> 
> Karl
> 
> On Wed, Feb 19, 2020 at 11:00 AM Bertrand Delacretaz
> <bd...@apache.org> wrote:
>>
>> Hi,
>>
>> On Wed, Feb 19, 2020 at 7:12 AM Carsten Ziegeler <cz...@apache.org> wrote:
>>> ...Having an option for compressing sounds good to me, but I get a little
>>> bit worried to support two extensions. It is transparent for the user of
>>> the archive..
>>
>> Karl seems to imply that having an uncompressed archive is required
>> for some use cases so I suppose there's a need for the "consumers" of
>> those archives to be able to specify that they require the
>> uncompressed version?
>>
>> If that's correct, a way of identifying those uncompressed versions is
>> required, IMHO.
>>
>> That can be the file extension, or maybe a Maven classifier?
>>
>> A common pattern used for tar archives is to add a suffix to indicate
>> compression, such as .tar.gz
>>
>> -Bertrand
> 
> 
> 

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Wed, Feb 19, 2020 at 11:06 AM Karl Pauls <ka...@gmail.com> wrote:
> ...if you want
> to provide both, an uncompressed and a compressed version, I guess you
> can just encode the difference in the classifier name...

That works for me, I was just thinking that a way to differentiate is required.

-Berrtrand

Re: [Feature Model] Feature Archive?

[Karl Pauls <ka...@gmail.com>]

I'm not sure we need to have a second extension. I would assume the
archive can be attached via a classifier anyways - hence, if you want
to provide both, an uncompressed and a compressed version, I guess you
can just encode the difference in the classifier name. Personally, I
would be fine with just making the extension be "zip" to begin with
since it sounds like that is all it is anyways (at least for now) but
I don't really feel strongly about it - if I can use it to create
uncompressed zips with everything in it I'm a happy camper already
:-).

regards,

Karl

On Wed, Feb 19, 2020 at 11:00 AM Bertrand Delacretaz
<bd...@apache.org> wrote:
>
> Hi,
>
> On Wed, Feb 19, 2020 at 7:12 AM Carsten Ziegeler <cz...@apache.org> wrote:
> > ...Having an option for compressing sounds good to me, but I get a little
> > bit worried to support two extensions. It is transparent for the user of
> > the archive..
>
> Karl seems to imply that having an uncompressed archive is required
> for some use cases so I suppose there's a need for the "consumers" of
> those archives to be able to specify that they require the
> uncompressed version?
>
> If that's correct, a way of identifying those uncompressed versions is
> required, IMHO.
>
> That can be the file extension, or maybe a Maven classifier?
>
> A common pattern used for tar archives is to add a suffix to indicate
> compression, such as .tar.gz
>
> -Bertrand



-- 
Karl Pauls
karlpauls@gmail.com

Re: [Feature Model] Feature Archive?

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Wed, Feb 19, 2020 at 7:12 AM Carsten Ziegeler <cz...@apache.org> wrote:
> ...Having an option for compressing sounds good to me, but I get a little
> bit worried to support two extensions. It is transparent for the user of
> the archive..

Karl seems to imply that having an uncompressed archive is required
for some use cases so I suppose there's a need for the "consumers" of
those archives to be able to specify that they require the
uncompressed version?

If that's correct, a way of identifying those uncompressed versions is
required, IMHO.

That can be the file extension, or maybe a Maven classifier?

A common pattern used for tar archives is to add a suffix to indicate
compression, such as .tar.gz

-Bertrand

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

Having an option for compressing sounds good to me, but I get a little 
bit worried to support two extensions. It is transparent for the user of 
the archive. As the archive usually contains binaries (which again might 
be compressed already), I'm not sure if there is a significant benefit 
in compressing the archive (in terms of the size of the archive).

Regards
Carsten

On 18.02.2020 11:23, Bertrand Delacretaz wrote:
> On Tue, Feb 18, 2020 at 11:00 AM Karl Pauls <ka...@gmail.com> wrote:
>> ...could we maybe make
>> it an option to create that zip file uncompressed?...
> 
> If you need this I would suggest using different extensions for the
> compressed and uncompressed formats - I suspect builds would add both
> to the Maven repository in this case?
> 
> So maybe .far and .zfar
> 
> -Bertrand (refraining hard form suggesting .near and .far)
> 

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[Bertrand Delacretaz <bd...@apache.org>]

On Tue, Feb 18, 2020 at 11:00 AM Karl Pauls <ka...@gmail.com> wrote:
> ...could we maybe make
> it an option to create that zip file uncompressed?...

If you need this I would suggest using different extensions for the
compressed and uncompressed formats - I suspect builds would add both
to the Maven repository in this case?

So maybe .far and .zfar

-Bertrand (refraining hard form suggesting .near and .far)

Re: [Feature Model] Feature Archive?

[Karl Pauls <ka...@gmail.com>]

If I understand the idea correctly you basically want to create a mvn
repo packages in zip file with the extension .far?

Assuming the above, I think that is a good idea - could we maybe make
it an option to create that zip file uncompressed? It would allow to
maybe use it with the springboot jar url handler without extracting
the artifacts first...

regards,

Karl

On Tue, Feb 18, 2020 at 7:08 AM Carsten Ziegeler <cz...@apache.org> wrote:
>
> If no one objects, I'll add an archive reader/writer to our feature IO
> model; the code is pretty similar to that one from the provisioning model.
>
> I suggest we use the extension "far" (no, there is no fourth letter) for
> the feature archives.
>
> Regards
> Carsten
>
> On 17.02.2020 12:22, Carsten Ziegeler wrote:
> > The archive acts like a distribution unit, instead of giving out the
> > model and requiring that the recipient also fetches all the individual
> > artifacts from "somewhere", you get everything in a single unit.
> >
> > So the archive is useful independent of installation.
> >
> > The installer is one way of using these archives at runtime. You could
> > also think of that you can launch an application based on a set of
> > archives. A tool could just extract all the archives into the file
> > system and then launch the app using the launcher.
> >
> > Regards
> > Carsten
> >
> > On 17.02.2020 11:24, Robert Munteanu wrote:
> >> Hi Carsten,
> >>
> >> On Mon, 2020-02-17 at 08:52 +0100, Carsten Ziegeler wrote:
> >>> Hi,
> >>>
> >>> with the older provisioning model we support an archive containing a
> >>> provisioning model and all contained artifacts ([1] and [2])
> >>> In addition we have a plugin for the OSGi installer which allows to
> >>> install such artifacts ([3]).
> >>>
> >>> The question is if we want to support something similar with the
> >>> feature
> >>> model?
> >>
> >> (snip)
> >>
> >> IIUC, the code creates an archive that can then be deployed to a
> >> running instance? Is there anything else that it can do?
> >>
> >> Thanks,
> >>
> >> Robert
> >>
> >
>
> --
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org



-- 
Karl Pauls
karlpauls@gmail.com

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

If no one objects, I'll add an archive reader/writer to our feature IO 
model; the code is pretty similar to that one from the provisioning model.

I suggest we use the extension "far" (no, there is no fourth letter) for 
the feature archives.

Regards
Carsten

On 17.02.2020 12:22, Carsten Ziegeler wrote:
> The archive acts like a distribution unit, instead of giving out the 
> model and requiring that the recipient also fetches all the individual 
> artifacts from "somewhere", you get everything in a single unit.
> 
> So the archive is useful independent of installation.
> 
> The installer is one way of using these archives at runtime. You could 
> also think of that you can launch an application based on a set of 
> archives. A tool could just extract all the archives into the file 
> system and then launch the app using the launcher.
> 
> Regards
> Carsten
> 
> On 17.02.2020 11:24, Robert Munteanu wrote:
>> Hi Carsten,
>>
>> On Mon, 2020-02-17 at 08:52 +0100, Carsten Ziegeler wrote:
>>> Hi,
>>>
>>> with the older provisioning model we support an archive containing a
>>> provisioning model and all contained artifacts ([1] and [2])
>>> In addition we have a plugin for the OSGi installer which allows to
>>> install such artifacts ([3]).
>>>
>>> The question is if we want to support something similar with the
>>> feature
>>> model?
>>
>> (snip)
>>
>> IIUC, the code creates an archive that can then be deployed to a
>> running instance? Is there anything else that it can do?
>>
>> Thanks,
>>
>> Robert
>>
> 

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

The archive acts like a distribution unit, instead of giving out the 
model and requiring that the recipient also fetches all the individual 
artifacts from "somewhere", you get everything in a single unit.

So the archive is useful independent of installation.

The installer is one way of using these archives at runtime. You could 
also think of that you can launch an application based on a set of 
archives. A tool could just extract all the archives into the file 
system and then launch the app using the launcher.

Regards
Carsten

On 17.02.2020 11:24, Robert Munteanu wrote:
> Hi Carsten,
> 
> On Mon, 2020-02-17 at 08:52 +0100, Carsten Ziegeler wrote:
>> Hi,
>>
>> with the older provisioning model we support an archive containing a
>> provisioning model and all contained artifacts ([1] and [2])
>> In addition we have a plugin for the OSGi installer which allows to
>> install such artifacts ([3]).
>>
>> The question is if we want to support something similar with the
>> feature
>> model?
> 
> (snip)
> 
> IIUC, the code creates an archive that can then be deployed to a
> running instance? Is there anything else that it can do?
> 
> Thanks,
> 
> Robert
> 

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org

Re: [Feature Model] Feature Archive?

[Robert Munteanu <ro...@apache.org>]

Hi Carsten,

On Mon, 2020-02-17 at 08:52 +0100, Carsten Ziegeler wrote:
> Hi,
> 
> with the older provisioning model we support an archive containing a 
> provisioning model and all contained artifacts ([1] and [2])
> In addition we have a plugin for the OSGi installer which allows to 
> install such artifacts ([3]).
> 
> The question is if we want to support something similar with the
> feature 
> model?

(snip)

IIUC, the code creates an archive that can then be deployed to a
running instance? Is there anything else that it can do?

Thanks,

Robert

Re: [Feature Model] Feature Archive?

[Carsten Ziegeler <cz...@apache.org>]

Hi,

I've implemented initial support for archives in feature.io. In addition 
I added a mojo to the slingfeature maven plugin which can be configured 
to create feature archives out of feature models.

I've also added an initial plugin to support features and feature 
archives with the OSGi installer to the whiteboard. But that code is 
untested yet :)

Most of it is very similar to what we did with the provisioning model 
but just changed to support the feature model.

Regards
Carsten

On 17.02.2020 08:52, Carsten Ziegeler wrote:
> Hi,
> 
> with the older provisioning model we support an archive containing a 
> provisioning model and all contained artifacts ([1] and [2])
> In addition we have a plugin for the OSGi installer which allows to 
> install such artifacts ([3]).
> 
> The question is if we want to support something similar with the feature 
> model?
> 
> Adjusting the code to support feature model based archives is very 
> trivial, question is just if we want to do it and how ?
> 
> Regards
> Carsten
> 
> 
> [1] 
> https://github.com/apache/sling-org-apache-sling-provisioning-model/blob/master/src/main/java/org/apache/sling/provisioning/model/io/ModelArchiveReader.java 
> 
> [2] 
> https://github.com/apache/sling-org-apache-sling-provisioning-model/blob/master/src/main/java/org/apache/sling/provisioning/model/io/ModelArchiveWriter.java 
> 
> [3] 
> https://github.com/apache/sling-org-apache-sling-installer-factory-model
> -- 
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org

-- 
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org