Recover image pulls by trying to run the container anyways.

[Markus Thoemmes <ma...@de.ibm.com>]

Dear OpenWhiskers,

when using Blackbox actions, whenever OpenWhisk creates a new container "docker pull" is executed to get the image for that container explicitly. It's implemented that way to guarantee new containers are always on the latest code-state that was pushed to Dockerhub.

Now for rapid prototyping that is a nice feature, for production workloads it might not be. Essentially this means that your production workload is now dependend on Dockerhub being up and running fine, even though your images might already have been pulled to all invoker machines necessary.

I therefore propose the following behavioral change:

1. Iff the user specifies "latest" as the image tag explicitly (or implicitly by not specifying a tag at all), the behavior will stay the same as above. If "docker pull" fails, the whole container bringup process fails and the user is given an error. That ensures, that for these workloads (which I consider development/prototyping) workloads, the user can deterministically know, that after an action update, each cold-start contains the very latest image or errors out.
2. For all actions with a tag different than "latest", they are considered production workloads. It is considered best-practice to not use "latest" for production images and expected that the user versions their images through the tags. OpenWhisk will still try to pull each time it creates a new container, for good measure. If the pull fails, the system will still try to "docker run" the container. If an image with that tag is already available locally, it will be picked up and the action runs just fine. This does not guarantee the latest code-state under all circumstances, but it's much more stable for applications in production.

An implementation of the proposal can be found here: https://github.com/apache/incubator-openwhisk/pull/3813

Feedback is very welcome :)

Cheers,
-m

Re: Recover image pulls by trying to run the container anyways.

[Rodric Rabbah <ro...@gmail.com>]

+1 to make the change - as noted this is for the better in terms of running docker actions in prod. 

-r

> On Jul 3, 2018, at 3:07 AM, Markus Thoemmes <ma...@de.ibm.com> wrote:
> 
> Hi,
> 
> Thanks for the clarification Rodric, wasn't aware of that. I think it still holds though that "latest" should mean "always pull and inform me if you failed" vs. other tags meaning "it's okay to swallow an occasional pull issue if that means my app runs just fine".
> 
> For future work, we could add the ability to pass digest information along with the image, which'd enable us to locally verify the image vs. having to do a network roundtrip.
> 
> Any concerns on the behavioral change initially proposed in the meantime? Should we hold back on the merge or can we go ahead?
> 
> Cheers,
> -m
> 

Re: Recover image pulls by trying to run the container anyways.

[Markus Thoemmes <ma...@de.ibm.com>]

Hi,

Thanks for the clarification Rodric, wasn't aware of that. I think it still holds though that "latest" should mean "always pull and inform me if you failed" vs. other tags meaning "it's okay to swallow an occasional pull issue if that means my app runs just fine".

For future work, we could add the ability to pass digest information along with the image, which'd enable us to locally verify the image vs. having to do a network roundtrip.

Any concerns on the behavioral change initially proposed in the meantime? Should we hold back on the merge or can we go ahead?

Cheers,
-m

Re: Recover image pulls by trying to run the container anyways.

[Carlos Santana <cs...@gmail.com>]

+1


On Wed, Jun 27, 2018 at 11:43 AM Rodric Rabbah <ro...@gmail.com> wrote:

> In general I think this is a good change - but minor nit that "latest" in
> docker doesn't mean pull the latest tag.
>
> -r
>
>
>
> On Wed, Jun 27, 2018 at 7:35 AM, Markus Thoemmes <
> markus.thoemmes@de.ibm.com
> > wrote:
>
> > Dear OpenWhiskers,
> >
> > when using Blackbox actions, whenever OpenWhisk creates a new container
> > "docker pull" is executed to get the image for that container explicitly.
> > It's implemented that way to guarantee new containers are always on the
> > latest code-state that was pushed to Dockerhub.
> >
> > Now for rapid prototyping that is a nice feature, for production
> workloads
> > it might not be. Essentially this means that your production workload is
> > now dependend on Dockerhub being up and running fine, even though your
> > images might already have been pulled to all invoker machines necessary.
> >
> > I therefore propose the following behavioral change:
> >
> > 1. Iff the user specifies "latest" as the image tag explicitly (or
> > implicitly by not specifying a tag at all), the behavior will stay the
> same
> > as above. If "docker pull" fails, the whole container bringup process
> fails
> > and the user is given an error. That ensures, that for these workloads
> > (which I consider development/prototyping) workloads, the user can
> > deterministically know, that after an action update, each cold-start
> > contains the very latest image or errors out.
> > 2. For all actions with a tag different than "latest", they are
> considered
> > production workloads. It is considered best-practice to not use "latest"
> > for production images and expected that the user versions their images
> > through the tags. OpenWhisk will still try to pull each time it creates a
> > new container, for good measure. If the pull fails, the system will still
> > try to "docker run" the container. If an image with that tag is already
> > available locally, it will be picked up and the action runs just fine.
> This
> > does not guarantee the latest code-state under all circumstances, but
> it's
> > much more stable for applications in production.
> >
> > An implementation of the proposal can be found here:
> > https://github.com/apache/incubator-openwhisk/pull/3813
> >
> > Feedback is very welcome :)
> >
> > Cheers,
> > -m
> >
> >
>

Re: Recover image pulls by trying to run the container anyways.

[Rodric Rabbah <ro...@gmail.com>]

In general I think this is a good change - but minor nit that "latest" in
docker doesn't mean pull the latest tag.

-r



On Wed, Jun 27, 2018 at 7:35 AM, Markus Thoemmes <markus.thoemmes@de.ibm.com
> wrote:

> Dear OpenWhiskers,
>
> when using Blackbox actions, whenever OpenWhisk creates a new container
> "docker pull" is executed to get the image for that container explicitly.
> It's implemented that way to guarantee new containers are always on the
> latest code-state that was pushed to Dockerhub.
>
> Now for rapid prototyping that is a nice feature, for production workloads
> it might not be. Essentially this means that your production workload is
> now dependend on Dockerhub being up and running fine, even though your
> images might already have been pulled to all invoker machines necessary.
>
> I therefore propose the following behavioral change:
>
> 1. Iff the user specifies "latest" as the image tag explicitly (or
> implicitly by not specifying a tag at all), the behavior will stay the same
> as above. If "docker pull" fails, the whole container bringup process fails
> and the user is given an error. That ensures, that for these workloads
> (which I consider development/prototyping) workloads, the user can
> deterministically know, that after an action update, each cold-start
> contains the very latest image or errors out.
> 2. For all actions with a tag different than "latest", they are considered
> production workloads. It is considered best-practice to not use "latest"
> for production images and expected that the user versions their images
> through the tags. OpenWhisk will still try to pull each time it creates a
> new container, for good measure. If the pull fails, the system will still
> try to "docker run" the container. If an image with that tag is already
> available locally, it will be picked up and the action runs just fine. This
> does not guarantee the latest code-state under all circumstances, but it's
> much more stable for applications in production.
>
> An implementation of the proposal can be found here:
> https://github.com/apache/incubator-openwhisk/pull/3813
>
> Feedback is very welcome :)
>
> Cheers,
> -m
>
>