Use of ACE for a consumer application

[Dan Gravell <el...@googlemail.com>]

Hi. I'm interested in using ACE in my product. I've always had trouble
managing the configuration of my OSGi bundles. I create a new release each
week with a set of bundles updated. Sometimes I add bundles, sometimes I
remove, but quite often something seems to go wrong with the configuration.

What I really want is something that can download the latest distribution
and install it for me, when the user says so. It feels like ACE can do this.

Unfortunately I cannot see much documentation on the site, so may I be
indulged some questions?

- How many targets can ACE scale to? I have over 2000 new users each month,
and probably about 20,000 active users

- How does target management work when users are behind common-or-garden
ISPs with dynamic IPs etc? I don't really care about what each individual
target is running, although it would be a nice feature to be able to
remotely manage a given target. But this would be exceptional (maybe
changing the logging on a particular user's target to diagnose some problem)

- How can this work with an installer? Ideally I'd like to install my
entire product when the user downloads it and not have a two stage
installation

- My app runs a Jetty web server where the Update UI lies... how does this
work with ACE? Do I contact the ACE server API, or interact with the agent
bundles co-located on the same OSGi container?

I use Felix and bndtools btw.

Dan

Re: Use of ACE for a consumer application

[Dan Gravell <el...@googlemail.com>]

Thanks Marcel. I'll begin having a play-around. I might progressively
introduce this to sections of my userbase (split by platform maybe) to see
how it goes.

Dan

On Wed, Mar 27, 2013 at 9:52 PM, Marcel Offermans <
marcel.offermans@luminis.nl> wrote:

> Hello Dan,
>
> On Mar 27, 2013, at 5:55 AM, Dan Gravell <el...@googlemail.com>
> wrote:
>
> > Hi Marcel...
> >
> >> - How many targets can ACE scale to? I have over 2000 new users each
> >> month,
> >>> and probably about 20,000 active users
> >>
> >> Hard to tell exactly, but we've setup ACE in such a way that you can not
> >> only create a single server for all the targets, but also create a
> master
> >> server with several "relay" servers (which act like proxies) to spread
> the
> >> load.
> >
> > It wasn't just the serving-up of distros I was wondering about... it was
> > stuff like the UI. One thing that took my notice was the targets column
> in
> > the admin UI (the one on the far right). I haven't actually tried it yet
> so
> > I don't know if that list is loaded lazily, or what. Does it show all
> > targets that ever existed?
>
> It is loaded lazily, but you are right that it's probably no longer
> practical to have such a long list of targets and browse through them.
>
> Probably in your case you don't care about the targets that much in the
> UI, so one thing you could do is simply not display them there at all. We
> have some code in there to, based on users and their rights, disable
> certain information.
>
> >> It also depends a bit on the number of updates, size of bundles and how
> >> often these targets poll for changes.
> >
> > A new distro is produced once a fortnight. The maximum size of a distro
> is
> > 35MB. I have control over the polling because I think I'll build the
> update
> > UI into my app... but I would call the first time the update UI is shown
> > and then when the user wanted to 'check for updates'.
>
> Ok, and probably you're not updating every bundle every time either,
> right? So from a bandwidth point of view that should still work.
>
> >>> - How does target management work when users are behind
> common-or-garden
> >>> ISPs with dynamic IPs etc? I don't really care about what each
> individual
> >>> target is running, although it would be a nice feature to be able to
> >>> remotely manage a given target. But this would be exceptional (maybe
> >>> changing the logging on a particular user's target to diagnose some
> >> problem)
> >>
> >> Right now, ACE keeps track of each individual target, but it can be
> setup
> >> so any new target registers automatically and gets a specific software
> >> distribution.
> >
> > What's this feature called and can I read about it anywhere?
>
> We have a bundle that can be found in this project:
>
> org.apache.ace.client.automation
>
> It automatically registers new targets as they appear. That's one part of
> the solution. The second part is making sure there is an association
> between a distribution that contains all the software you want to deploy to
> such new targets and the new targets. Associations in ACE have a left hand
> and right hand side. Both can use filter conditions to match entities. If
> you normally create them using the web UI, you create 1 to 1 associations.
> If you use the REST endpoint, you have more flexibility and can create an
> assocation that will have a filter that matches all targets.
>
> There is some documentation about that here:
>
> http://ace.apache.org/user-doc/restapi.html
>
> You should try to create a "distribution2target" with a leftEndpoint that
> matches your distribution and a rightEndpoint that is an LDAP filter,
> something like: (name=*)
>
> >> ACE uses a management agent, which polls the server using HTTP. So
> dynamic
> >> IPs are no problem, and firewalls usually also not.
> >
> > When you say 'polls'... any recurring 'poll' can be switched off, right?
> I
> > just want to check for a new distro when the user asks for it (for
> now... I
> > suppose a Chrome-like dream of fully automatic updates is one to
> consider).
>
> Yes, see below for the link to the docs about this.
>
> >>> - How can this work with an installer? Ideally I'd like to install my
> >>> entire product when the user downloads it and not have a two stage
> >>> installation
> >>
> >> That is currently not supported, unfortunately, but it is definitely an
> >> interesting use case that we should explore. We actually do have some
> >> support for installing our deployment packages from a local filesystem,
> but
> >> we cannot at this point make a seamless switch from local installation
> to
> >> remote installation. I would not mind at all putting that on the roadmap
> >> though.
> >
> > My concern with two stage installers is that the user will become
> > frustrated with another wait while the app initialises itself and
> downloads
> > its latest code. Maybe it's a false fear. Maybe (I use IzPack) I can just
> > write a plugin to the installer which contacts the ACE server...
>
> I think you have a valid point, and it's worth seeing if ACE can support
> this in the future.
>
> > Apps with ACE agents work fine offline, right? I don't want a SimCity
> style
> > situation on my hands ;)
>
> Yes, they do work off-line, no problems there.
>
> >>> - My app runs a Jetty web server where the Update UI lies... how does
> >> this
> >>> work with ACE? Do I contact the ACE server API, or interact with the
> >> agent
> >>> bundles co-located on the same OSGi container?
> >>
> >> The management agent by default has an embedded scheduler that
> >> periodically polls the server. Instead of that scheduler, you can also
> >> directly invoke the update task from OSGi (it is a service). So your
> update
> >> UI could do that, and disable the scheduler.
> >
> > Any links to docs for this? Or is this 'read the code'?
>
> There is documentation for this on the website:
>
> http://ace.apache.org/dev-doc/design/ace-deployment-strategies.html
>
> >> I'm sure these answers have triggered more questions, so don't hesitate
> to
> >> keep asking! :)
> >>
> >
> > Will do, have done :)
> >
> > Assume you're at EclipseCon, hope it's going well.
>
> Yes, talking to lots of interesting people here, thanks!
>
> Greetings, Marcel
>
>

Re: Use of ACE for a consumer application

[Marcel Offermans <ma...@luminis.nl>]

Hello Dan,

On Mar 27, 2013, at 5:55 AM, Dan Gravell <el...@googlemail.com> wrote:

> Hi Marcel...
> 
>> - How many targets can ACE scale to? I have over 2000 new users each
>> month,
>>> and probably about 20,000 active users
>> 
>> Hard to tell exactly, but we've setup ACE in such a way that you can not
>> only create a single server for all the targets, but also create a master
>> server with several "relay" servers (which act like proxies) to spread the
>> load.
> 
> It wasn't just the serving-up of distros I was wondering about... it was
> stuff like the UI. One thing that took my notice was the targets column in
> the admin UI (the one on the far right). I haven't actually tried it yet so
> I don't know if that list is loaded lazily, or what. Does it show all
> targets that ever existed?

It is loaded lazily, but you are right that it's probably no longer practical to have such a long list of targets and browse through them.

Probably in your case you don't care about the targets that much in the UI, so one thing you could do is simply not display them there at all. We have some code in there to, based on users and their rights, disable certain information.

>> It also depends a bit on the number of updates, size of bundles and how
>> often these targets poll for changes.
> 
> A new distro is produced once a fortnight. The maximum size of a distro is
> 35MB. I have control over the polling because I think I'll build the update
> UI into my app... but I would call the first time the update UI is shown
> and then when the user wanted to 'check for updates'.

Ok, and probably you're not updating every bundle every time either, right? So from a bandwidth point of view that should still work.

>>> - How does target management work when users are behind common-or-garden
>>> ISPs with dynamic IPs etc? I don't really care about what each individual
>>> target is running, although it would be a nice feature to be able to
>>> remotely manage a given target. But this would be exceptional (maybe
>>> changing the logging on a particular user's target to diagnose some
>> problem)
>> 
>> Right now, ACE keeps track of each individual target, but it can be setup
>> so any new target registers automatically and gets a specific software
>> distribution.
> 
> What's this feature called and can I read about it anywhere?

We have a bundle that can be found in this project:

org.apache.ace.client.automation

It automatically registers new targets as they appear. That's one part of the solution. The second part is making sure there is an association between a distribution that contains all the software you want to deploy to such new targets and the new targets. Associations in ACE have a left hand and right hand side. Both can use filter conditions to match entities. If you normally create them using the web UI, you create 1 to 1 associations. If you use the REST endpoint, you have more flexibility and can create an assocation that will have a filter that matches all targets.

There is some documentation about that here:

http://ace.apache.org/user-doc/restapi.html

You should try to create a "distribution2target" with a leftEndpoint that matches your distribution and a rightEndpoint that is an LDAP filter, something like: (name=*)

>> ACE uses a management agent, which polls the server using HTTP. So dynamic
>> IPs are no problem, and firewalls usually also not.
> 
> When you say 'polls'... any recurring 'poll' can be switched off, right? I
> just want to check for a new distro when the user asks for it (for now... I
> suppose a Chrome-like dream of fully automatic updates is one to consider).

Yes, see below for the link to the docs about this.

>>> - How can this work with an installer? Ideally I'd like to install my
>>> entire product when the user downloads it and not have a two stage
>>> installation
>> 
>> That is currently not supported, unfortunately, but it is definitely an
>> interesting use case that we should explore. We actually do have some
>> support for installing our deployment packages from a local filesystem, but
>> we cannot at this point make a seamless switch from local installation to
>> remote installation. I would not mind at all putting that on the roadmap
>> though.
> 
> My concern with two stage installers is that the user will become
> frustrated with another wait while the app initialises itself and downloads
> its latest code. Maybe it's a false fear. Maybe (I use IzPack) I can just
> write a plugin to the installer which contacts the ACE server...

I think you have a valid point, and it's worth seeing if ACE can support this in the future.

> Apps with ACE agents work fine offline, right? I don't want a SimCity style
> situation on my hands ;)

Yes, they do work off-line, no problems there.

>>> - My app runs a Jetty web server where the Update UI lies... how does
>> this
>>> work with ACE? Do I contact the ACE server API, or interact with the
>> agent
>>> bundles co-located on the same OSGi container?
>> 
>> The management agent by default has an embedded scheduler that
>> periodically polls the server. Instead of that scheduler, you can also
>> directly invoke the update task from OSGi (it is a service). So your update
>> UI could do that, and disable the scheduler.
> 
> Any links to docs for this? Or is this 'read the code'?

There is documentation for this on the website:

http://ace.apache.org/dev-doc/design/ace-deployment-strategies.html

>> I'm sure these answers have triggered more questions, so don't hesitate to
>> keep asking! :)
>> 
> 
> Will do, have done :)
> 
> Assume you're at EclipseCon, hope it's going well.

Yes, talking to lots of interesting people here, thanks!

Greetings, Marcel

Re: Use of ACE for a consumer application

[Dan Gravell <el...@googlemail.com>]

Hi Marcel...

> - How many targets can ACE scale to? I have over 2000 new users each
> month,
> > and probably about 20,000 active users
>
> Hard to tell exactly, but we've setup ACE in such a way that you can not
> only create a single server for all the targets, but also create a master
> server with several "relay" servers (which act like proxies) to spread the
> load.
>

It wasn't just the serving-up of distros I was wondering about... it was
stuff like the UI. One thing that took my notice was the targets column in
the admin UI (the one on the far right). I haven't actually tried it yet so
I don't know if that list is loaded lazily, or what. Does it show all
targets that ever existed?


> It also depends a bit on the number of updates, size of bundles and how
> often these targets poll for changes.
>

A new distro is produced once a fortnight. The maximum size of a distro is
35MB. I have control over the polling because I think I'll build the update
UI into my app... but I would call the first time the update UI is shown
and then when the user wanted to 'check for updates'.


> > - How does target management work when users are behind common-or-garden
> > ISPs with dynamic IPs etc? I don't really care about what each individual
> > target is running, although it would be a nice feature to be able to
> > remotely manage a given target. But this would be exceptional (maybe
> > changing the logging on a particular user's target to diagnose some
> problem)
>
> Right now, ACE keeps track of each individual target, but it can be setup
> so any new target registers automatically and gets a specific software
> distribution.
>

What's this feature called and can I read about it anywhere?


> ACE uses a management agent, which polls the server using HTTP. So dynamic
> IPs are no problem, and firewalls usually also not.
>

When you say 'polls'... any recurring 'poll' can be switched off, right? I
just want to check for a new distro when the user asks for it (for now... I
suppose a Chrome-like dream of fully automatic updates is one to consider).


> > - How can this work with an installer? Ideally I'd like to install my
> > entire product when the user downloads it and not have a two stage
> > installation
>
> That is currently not supported, unfortunately, but it is definitely an
> interesting use case that we should explore. We actually do have some
> support for installing our deployment packages from a local filesystem, but
> we cannot at this point make a seamless switch from local installation to
> remote installation. I would not mind at all putting that on the roadmap
> though.
>

My concern with two stage installers is that the user will become
frustrated with another wait while the app initialises itself and downloads
its latest code. Maybe it's a false fear. Maybe (I use IzPack) I can just
write a plugin to the installer which contacts the ACE server...

Apps with ACE agents work fine offline, right? I don't want a SimCity style
situation on my hands ;)


> > - My app runs a Jetty web server where the Update UI lies... how does
> this
> > work with ACE? Do I contact the ACE server API, or interact with the
> agent
> > bundles co-located on the same OSGi container?
>
> The management agent by default has an embedded scheduler that
> periodically polls the server. Instead of that scheduler, you can also
> directly invoke the update task from OSGi (it is a service). So your update
> UI could do that, and disable the scheduler.
>

Any links to docs for this? Or is this 'read the code'?


> I'm sure these answers have triggered more questions, so don't hesitate to
> keep asking! :)
>

Will do, have done :)

Assume you're at EclipseCon, hope it's going well.

Dan

Re: Use of ACE for a consumer application

[Marcel Offermans <ma...@luminis.nl>]

Hello Dan,

On Mar 26, 2013, at 7:41 AM, Dan Gravell <el...@googlemail.com> wrote:

> Hi. I'm interested in using ACE in my product. I've always had trouble
> managing the configuration of my OSGi bundles. I create a new release each
> week with a set of bundles updated. Sometimes I add bundles, sometimes I
> remove, but quite often something seems to go wrong with the configuration.
> 
> What I really want is something that can download the latest distribution
> and install it for me, when the user says so. It feels like ACE can do this.

Yes.

> Unfortunately I cannot see much documentation on the site, so may I be
> indulged some questions?

Sure.

> - How many targets can ACE scale to? I have over 2000 new users each month,
> and probably about 20,000 active users

Hard to tell exactly, but we've setup ACE in such a way that you can not only create a single server for all the targets, but also create a master server with several "relay" servers (which act like proxies) to spread the load.

It also depends a bit on the number of updates, size of bundles and how often these targets poll for changes.

> - How does target management work when users are behind common-or-garden
> ISPs with dynamic IPs etc? I don't really care about what each individual
> target is running, although it would be a nice feature to be able to
> remotely manage a given target. But this would be exceptional (maybe
> changing the logging on a particular user's target to diagnose some problem)

Right now, ACE keeps track of each individual target, but it can be setup so any new target registers automatically and gets a specific software distribution.

ACE uses a management agent, which polls the server using HTTP. So dynamic IPs are no problem, and firewalls usually also not.

> - How can this work with an installer? Ideally I'd like to install my
> entire product when the user downloads it and not have a two stage
> installation

That is currently not supported, unfortunately, but it is definitely an interesting use case that we should explore. We actually do have some support for installing our deployment packages from a local filesystem, but we cannot at this point make a seamless switch from local installation to remote installation. I would not mind at all putting that on the roadmap though.

> - My app runs a Jetty web server where the Update UI lies... how does this
> work with ACE? Do I contact the ACE server API, or interact with the agent
> bundles co-located on the same OSGi container?

The management agent by default has an embedded scheduler that periodically polls the server. Instead of that scheduler, you can also directly invoke the update task from OSGi (it is a service). So your update UI could do that, and disable the scheduler.

I'm sure these answers have triggered more questions, so don't hesitate to keep asking! :)

Greetings, Marcel