Don't differentiate between blackbox and whitebox images

[Christian Bickel <cb...@apache.org>]

Hi,

in the past we divided the invokers into blackbox and whitebox invokers.
This has the effect that one of these two types could be overloaded while
the other type still has free capacity.

Does anyone see issues on using every invoker for everything?
Or does anyone see any action items that need to be addressed before we
invoke every action-type on every invoker?

If not I'll open a PR to not differentiate between these types anymore.

Greetings
Christian Bickel

Re: Don't differentiate between blackbox and whitebox images

[Sven Lange-Last <sv...@de.ibm.com>]

I agree with Rodric. In the past, our assumption was that the risk 
associated with blackbox invocations is higher than with managed 
invocations. That's why we used two separate invoker pools such that the 
possible problems with blackbox invocations won't affect any managed 
invokers and thus, managed invocations.

We need to re-evaluate whether we still see the same risk as in the past 
and whether we see ways to address possible problems.

For those who are willing to accept certain risks, Rodric's proposal of 
providing flexibility via configuration settings may be the right 
approach. Apparently, the requirements depend on the type of OW 
environment you run. For a large production environment supporting a large 
number of different tenants requirements are different than for your local 
development environment.

I think we have (at least) these risks:

=> Docker images consume too much disk space so that no more new images 
can be pulled / no new containers can be created.

A limited set of Docker images is required to support managed invocations. 
The situation is different for blackbox invocations. Potentially each 
blackbox invocation may lead to a pull of an additional Docker image so 
that the disk volume storing images and containers may fill up. At the 
moment, there are no restrictions on the number and size of blackbox 
images on an invoker. So invokers running blackbox actions have a higher 
risk for outages.

OW does not have any image cleanup strategy and a limited container 
cleanup strategy. Kubernetes has a garbage collection approach [1]. We may 
add a cleanup strategy to OW as well or users need their own cleanup 
approach outside of OW.

=> Managed runtime images have more control.

All managed runtimes containers run OW code up to the point where the 
/init REST endpoint can be used to inject end user action code. So we can 
prepare the environment and accept the initial socket connection from the 
invoker. Our timeout concept bases these socket connections.

Blackbox containers have more control and can decide to not even open a 
listening socket. It's harder for the invoker to deal with such a 
non-cooperating container and prevent the container from consuming 
resources.

=> Blackbox images can bring much more user code than managed actions.

It's easier to hide malicious code or tools in a blackbox image.

=> Blackbox image pulls keep container infrastructure busy.

As already mentioned, each blackbox invocation may lead to a pull of an 
additional Docker image. These pulls consume CPU and network resources 
impacting other invocations. The pull mechanism could even be used to 
attack Docker Hub.


Any other thoughts or feedback on the risks I described?


Regards, Sven.


[1] 
https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

Rodric Rabbah <ro...@gmail.com> wrote on 2018/11/28 13:09:12:

> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 2018/11/28 13:09
> Subject: Re: Don't differentiate between blackbox and whitebox images
> 
> I think it would be a mistake to blindly merge two container pools -
> docker actions as run today are pulled as needed and can take a long
> time, they’re subject to different kinds of attacks, and can affect 
> performance of other tenants (higher noisy neighbors). 
> 
> If you want to allow more overlap, it should be at least done as a 
> configuration parameter which allows different pools to continue to 
> exist (note that for small N as in local dev they’re allowed to 
> overlap today). Or, used for overflow only as a way of providing 
> more elasticity.
> 
> -r
> 
> > On Nov 28, 2018, at 5:56 AM, Christian Bickel <cb...@apache.org> 
wrote:
> > 
> > Hi,
> > 
> > in the past we divided the invokers into blackbox and whitebox 
invokers.
> > This has the effect that one of these two types could be overloaded 
while
> > the other type still has free capacity.
> > 
> > Does anyone see issues on using every invoker for everything?
> > Or does anyone see any action items that need to be addressed before 
we
> > invoke every action-type on every invoker?
> > 
> > If not I'll open a PR to not differentiate between these types 
anymore.
> > 
> > Greetings
> > Christian Bickel

Re: Don't differentiate between blackbox and whitebox images

[Carlos Santana <cs...@gmail.com>]

By configuration you can have different combinations.


Blackbox actions on blackbox invoker only, and regular actions on normal
invoker types only (ie. today)
Blackbox actions on blackbox invoker only, and regular actions on any
invoker type
Any actions on any invoker type


On Wed, Nov 28, 2018 at 7:09 AM Rodric Rabbah <ro...@gmail.com> wrote:

> I think it would be a mistake to blindly merge two container pools -
> docker actions as run today are pulled as needed and can take a long time,
> they’re subject to different kinds of attacks, and can affect performance
> of other tenants (higher noisy neighbors).
>
> If you want to allow more overlap, it should be at least done as a
> configuration parameter which allows different pools to continue to exist
> (note that for small N as in local dev they’re allowed to overlap today).
> Or, used for overflow only as a way of providing more elasticity.
>
> -r
>
> > On Nov 28, 2018, at 5:56 AM, Christian Bickel <cb...@apache.org>
> wrote:
> >
> > Hi,
> >
> > in the past we divided the invokers into blackbox and whitebox invokers.
> > This has the effect that one of these two types could be overloaded while
> > the other type still has free capacity.
> >
> > Does anyone see issues on using every invoker for everything?
> > Or does anyone see any action items that need to be addressed before we
> > invoke every action-type on every invoker?
> >
> > If not I'll open a PR to not differentiate between these types anymore.
> >
> > Greetings
> > Christian Bickel
>


-- 
Carlos Santana
<cs...@gmail.com>

Re: Don't differentiate between blackbox and whitebox images

[David P Grove <gr...@us.ibm.com>]

Rodric Rabbah <ro...@gmail.com> wrote on 11/28/2018 07:09:12 AM:
>
> I think it would be a mistake to blindly merge two container pools -
> docker actions as run today are pulled as needed and can take a long
> time, they’re subject to different kinds of attacks, and can affect
> performance of other tenants (higher noisy neighbors).
>
> If you want to allow more overlap, it should be at least done as a
> configuration parameter which allows different pools to continue to
> exist (note that for small N as in local dev they’re allowed to
> overlap today). Or, used for overflow only as a way of providing
> more elasticity.
>

Note that if we are using the KubernetesContainerFactory (or I believe the
MesosContainerFactory) the separation at the invoker level is not really
necessarily effective at avoiding noisy neighbors since there is no longer
a 1:1 match from invoker to docker daemons.

--dave

Re: Don't differentiate between blackbox and whitebox images

[Rodric Rabbah <ro...@gmail.com>]

I think it would be a mistake to blindly merge two container pools - docker actions as run today are pulled as needed and can take a long time, they’re subject to different kinds of attacks, and can affect performance of other tenants (higher noisy neighbors). 

If you want to allow more overlap, it should be at least done as a configuration parameter which allows different pools to continue to exist (note that for small N as in local dev they’re allowed to overlap today). Or, used for overflow only as a way of providing more elasticity.

-r

> On Nov 28, 2018, at 5:56 AM, Christian Bickel <cb...@apache.org> wrote:
> 
> Hi,
> 
> in the past we divided the invokers into blackbox and whitebox invokers.
> This has the effect that one of these two types could be overloaded while
> the other type still has free capacity.
> 
> Does anyone see issues on using every invoker for everything?
> Or does anyone see any action items that need to be addressed before we
> invoke every action-type on every invoker?
> 
> If not I'll open a PR to not differentiate between these types anymore.
> 
> Greetings
> Christian Bickel