You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Avinash Sridharan <av...@mesosphere.io> on 2016/12/07 23:01:20 UTC
Adding a reload end-point to `network/cni` isolator
@adam @vinod Starting to work on
https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
The goal is to allow the `network/cni` isolator to load CNI configs without
the need for agent restarts. Had a discussion with @jieyu and the solution
we came up with was for the `network/cni` isolator to expose an endpoint
called `reload`. The endpoint will accept `POST` requests (with an empty
body), which will trigger the `network/cni` isolator to reload the CNI
configs present in the `network_cni_config_dir`. On a successful `reload`
the `network/cni` isolator will respond with an empty HTTP response. Wanted
to run this by you guys to understand the implications on authn/authz and
if this is the right place (the `network/cni` isolator) to host this
endponit?
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
On Wed, Dec 7, 2016 at 3:26 PM, Vinod Kone <vi...@apache.org> wrote:
> We are trying to move away from un-versioned REST endpoints like "/reload"
> in favor of versioned endpoint(s) that take calls. We don't have a good
> story yet for endpoints that are not hosted by Master and Agent actors. So,
> at first blush, I'm wary of this solution.
>
> Ok, this definitely throws a spanner to the idea of having a reload
endpoint.
> - Why not scan the config directory every time? Are you trying to avoid the
> > speed hit from disk reads?
> >
>
> Why not do this? If a network config doesn't exist in memory, refresh
> configs by reading from disk and check again; basically acting like a
> cache. Seems straight forward enough?
>
This does seem feasible. We were trying to avoid unnecessary reads . For
example a framework continuously launches containers on a non-existent
network (due to operator error or malicious intent). This would result in
quite a few unnecessary reads. However, if the reload endpoint is not a
possibility than this would be the route we have to take.
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Vinod Kone <vi...@apache.org>.
We are trying to move away from un-versioned REST endpoints like "/reload"
in favor of versioned endpoint(s) that take calls. We don't have a good
story yet for endpoints that are not hosted by Master and Agent actors. So,
at first blush, I'm wary of this solution.
- Why not scan the config directory every time? Are you trying to avoid the
> speed hit from disk reads?
>
Why not do this? If a network config doesn't exist in memory, refresh
configs by reading from disk and check again; basically acting like a
cache. Seems straight forward enough?
Re: Adding a reload end-point to `network/cni` isolator
Posted by Alex Rukletsov <al...@mesosphere.io>.
Whatever the solution will be, it will be great to stay consistent. It
looks like updating configuration during agent lifetime is a typical task,
hence having a "standard approach" would be great. Agent Whitelist and ACLS
come to my mind.
On 8 Dec 2016 7:24 am, "Avinash Sridharan" <av...@mesosphere.io> wrote:
> Valid point.
>
> Looks like this solution is turning out to much more cleaner than I
> expected :).
>
> Thanks Daniel, Qian and Vinod.
>
> This was helpful.
>
> On Wed, Dec 7, 2016 at 6:22 PM, Qian Zhang <zh...@gmail.com> wrote:
>
> > Why does "delete" need an agent restart? I think operators can just
> delete
> > the CNI network configuration file from "--network_cni_config_dir" at any
> > time they want, and later when a framework tries to launch a container to
> > that deleted CNI network, CNI isolator will find that network is in its
> > cache but not in the disk, so it can fail framework's request and remove
> > that CNI network from its cache. So it is kind of lazy delete in cache.
> >
> >
> > Thanks,
> > Qian Zhang
> >
> > On Thu, Dec 8, 2016 at 8:12 AM, Avinash Sridharan <avinash@mesosphere.io
> >
> > wrote:
> >
> > > On Wed, Dec 7, 2016 at 4:07 PM, Daniel Osborne <da...@tigera.io> wrote:
> > >
> > > > For the record, we already support a). Qian explains it here:
> > > > https://issues.apache.org/jira/browse/MESOS-6567?
> > > > focusedCommentId=15652501&page=com.atlassian.jira.
> > > > plugin.system.issuetabpanels:comment-tabpanel#comment-15652501
> > > >
> > > > You are correct. We don't store the config in-memory just the `name`.
> > So
> > > we will be reading the config every time we launch a new container. So
> > > looks like "delete" is the only operation that will need an agent
> > restart.
> > >
> > > >
> > > > On Wed, Dec 7, 2016 at 4:02 PM, Avinash Sridharan <
> > avinash@mesosphere.io
> > > >
> > > > wrote:
> > > >
> > > > > Thinking about the solution of treating the CNI config as an
> > in-memory
> > > > > cache and doing disk reads on failures I see two problems:
> > > > > a) We won't be able to support modifications to CNI networks. Since
> > > > > modification to existing networks won't generate a miss.
> > > > > b) We won't be able to support deletion of CNI networks.
> > > > >
> > > > > The two operations above will still need an agent restart.
> > > > >
> > > > > On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <
> > > avinash@mesosphere.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <
> > > > avinash@mesosphere.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io>
> > > wrote:
> > > > > >>
> > > > > >>> Chiming in since I raised an identical issue a few weeks back:
> > > > > >>> https://issues.apache.org/jira/browse/MESOS-6567
> > > > > >>>
> > > > > >>> The proposed endpoint solution sounds plausible. However I'd
> like
> > > to
> > > > > >>> explore if it solves the use case I raised my issue for. I was
> > > trying
> > > > > to
> > > > > >>> create a Mesos framework that adds new CNI networks. But [IIRC]
> > the
> > > > > Agent
> > > > > >>> API can't be reached from a Mesos Executor instance since the
> > Agent
> > > > > could
> > > > > >>> be listening on a non-default port, or on any of its IPs. The
> > > > executor
> > > > > >>> instance doesn't know that information, so after it installs
> the
> > > > > plugin,
> > > > > >>> it
> > > > > >>> won't know how to reach that new reload endpoint.
> > > > > >>>
> > > > > >>
> > > > > >> Just trying to understand the problem you are alluding to here.
> > The
> > > > > >> executor needs to register with the agent in order to launch the
> > > > > container,
> > > > > >> so it should have reachability to the agent, and hence the
> > endpoint?
> > > > > >>
> > > > > >>
> > > > > >>> - Is there a reliable way to reach the reload endpoint from a
> > > > default
> > > > > >>> executor instance?
> > > > > >>> - Why not scan the config directory every time? Are you trying
> to
> > > > avoid
> > > > > >>> the
> > > > > >>> speed hit from disk reads?
> > > > > >>>
> > > > > >> By scan the config directory every time, do you mean run a timer
> > > that
> > > > > >> will periodically scan the config directory and keep updating
> the
> > > > > configs.
> > > > > >> This is feasible. The only problem is that the point at which
> the
> > > > > operator
> > > > > >> write the config and the point at which the network will be
> > > available
> > > > > for
> > > > > >> container launch will not be deterministic. The behavior would
> be
> > > much
> > > > > >> cleaner if we can make it deterministic.
> > > > > >>
> > > > > >
> > > > > > Daniel, ignore this comment. I think you were referring to using
> > the
> > > > disc
> > > > > > as a cache as Vinod had pointed out. I misread your suggestion.
> > > > > >
> > > > > >> Best,
> > > > > >>> -Dan
> > > > > >>>
> > > > > >>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <
> > > > > avinash@mesosphere.io
> > > > > >>> >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > @adam @vinod Starting to work on
> > > > > >>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some
> > > > inputs.
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > The goal is to allow the `network/cni` isolator to load CNI
> > > configs
> > > > > >>> without
> > > > > >>> > the need for agent restarts. Had a discussion with @jieyu and
> > the
> > > > > >>> solution
> > > > > >>> > we came up with was for the `network/cni` isolator to expose
> an
> > > > > >>> endpoint
> > > > > >>> > called `reload`. The endpoint will accept `POST` requests
> (with
> > > an
> > > > > >>> empty
> > > > > >>> > body), which will trigger the `network/cni` isolator to
> reload
> > > the
> > > > > CNI
> > > > > >>> > configs present in the `network_cni_config_dir`. On a
> > successful
> > > > > >>> `reload`
> > > > > >>> > the `network/cni` isolator will respond with an empty HTTP
> > > > response.
> > > > > >>> Wanted
> > > > > >>> > to run this by you guys to understand the implications on
> > > > authn/authz
> > > > > >>> and
> > > > > >>> > if this is the right place (the `network/cni` isolator) to
> host
> > > > this
> > > > > >>> > endponit?
> > > > > >>> >
> > > > > >>> > --
> > > > > >>> > Avinash Sridharan, Mesosphere
> > > > > >>> > +1 (323) 702 5245
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Avinash Sridharan, Mesosphere
> > > > > >> +1 (323) 702 5245
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Avinash Sridharan, Mesosphere
> > > > > > +1 (323) 702 5245
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Avinash Sridharan, Mesosphere
> > > > > +1 (323) 702 5245
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Avinash Sridharan, Mesosphere
> > > +1 (323) 702 5245
> > >
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
Valid point.
Looks like this solution is turning out to much more cleaner than I
expected :).
Thanks Daniel, Qian and Vinod.
This was helpful.
On Wed, Dec 7, 2016 at 6:22 PM, Qian Zhang <zh...@gmail.com> wrote:
> Why does "delete" need an agent restart? I think operators can just delete
> the CNI network configuration file from "--network_cni_config_dir" at any
> time they want, and later when a framework tries to launch a container to
> that deleted CNI network, CNI isolator will find that network is in its
> cache but not in the disk, so it can fail framework's request and remove
> that CNI network from its cache. So it is kind of lazy delete in cache.
>
>
> Thanks,
> Qian Zhang
>
> On Thu, Dec 8, 2016 at 8:12 AM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > On Wed, Dec 7, 2016 at 4:07 PM, Daniel Osborne <da...@tigera.io> wrote:
> >
> > > For the record, we already support a). Qian explains it here:
> > > https://issues.apache.org/jira/browse/MESOS-6567?
> > > focusedCommentId=15652501&page=com.atlassian.jira.
> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15652501
> > >
> > > You are correct. We don't store the config in-memory just the `name`.
> So
> > we will be reading the config every time we launch a new container. So
> > looks like "delete" is the only operation that will need an agent
> restart.
> >
> > >
> > > On Wed, Dec 7, 2016 at 4:02 PM, Avinash Sridharan <
> avinash@mesosphere.io
> > >
> > > wrote:
> > >
> > > > Thinking about the solution of treating the CNI config as an
> in-memory
> > > > cache and doing disk reads on failures I see two problems:
> > > > a) We won't be able to support modifications to CNI networks. Since
> > > > modification to existing networks won't generate a miss.
> > > > b) We won't be able to support deletion of CNI networks.
> > > >
> > > > The two operations above will still need an agent restart.
> > > >
> > > > On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <
> > avinash@mesosphere.io
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <
> > > avinash@mesosphere.io
> > > > >
> > > > > wrote:
> > > > >
> > > > >>
> > > > >>
> > > > >> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io>
> > wrote:
> > > > >>
> > > > >>> Chiming in since I raised an identical issue a few weeks back:
> > > > >>> https://issues.apache.org/jira/browse/MESOS-6567
> > > > >>>
> > > > >>> The proposed endpoint solution sounds plausible. However I'd like
> > to
> > > > >>> explore if it solves the use case I raised my issue for. I was
> > trying
> > > > to
> > > > >>> create a Mesos framework that adds new CNI networks. But [IIRC]
> the
> > > > Agent
> > > > >>> API can't be reached from a Mesos Executor instance since the
> Agent
> > > > could
> > > > >>> be listening on a non-default port, or on any of its IPs. The
> > > executor
> > > > >>> instance doesn't know that information, so after it installs the
> > > > plugin,
> > > > >>> it
> > > > >>> won't know how to reach that new reload endpoint.
> > > > >>>
> > > > >>
> > > > >> Just trying to understand the problem you are alluding to here.
> The
> > > > >> executor needs to register with the agent in order to launch the
> > > > container,
> > > > >> so it should have reachability to the agent, and hence the
> endpoint?
> > > > >>
> > > > >>
> > > > >>> - Is there a reliable way to reach the reload endpoint from a
> > > default
> > > > >>> executor instance?
> > > > >>> - Why not scan the config directory every time? Are you trying to
> > > avoid
> > > > >>> the
> > > > >>> speed hit from disk reads?
> > > > >>>
> > > > >> By scan the config directory every time, do you mean run a timer
> > that
> > > > >> will periodically scan the config directory and keep updating the
> > > > configs.
> > > > >> This is feasible. The only problem is that the point at which the
> > > > operator
> > > > >> write the config and the point at which the network will be
> > available
> > > > for
> > > > >> container launch will not be deterministic. The behavior would be
> > much
> > > > >> cleaner if we can make it deterministic.
> > > > >>
> > > > >
> > > > > Daniel, ignore this comment. I think you were referring to using
> the
> > > disc
> > > > > as a cache as Vinod had pointed out. I misread your suggestion.
> > > > >
> > > > >> Best,
> > > > >>> -Dan
> > > > >>>
> > > > >>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <
> > > > avinash@mesosphere.io
> > > > >>> >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > @adam @vinod Starting to work on
> > > > >>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some
> > > inputs.
> > > > >>> >
> > > > >>> >
> > > > >>> > The goal is to allow the `network/cni` isolator to load CNI
> > configs
> > > > >>> without
> > > > >>> > the need for agent restarts. Had a discussion with @jieyu and
> the
> > > > >>> solution
> > > > >>> > we came up with was for the `network/cni` isolator to expose an
> > > > >>> endpoint
> > > > >>> > called `reload`. The endpoint will accept `POST` requests (with
> > an
> > > > >>> empty
> > > > >>> > body), which will trigger the `network/cni` isolator to reload
> > the
> > > > CNI
> > > > >>> > configs present in the `network_cni_config_dir`. On a
> successful
> > > > >>> `reload`
> > > > >>> > the `network/cni` isolator will respond with an empty HTTP
> > > response.
> > > > >>> Wanted
> > > > >>> > to run this by you guys to understand the implications on
> > > authn/authz
> > > > >>> and
> > > > >>> > if this is the right place (the `network/cni` isolator) to host
> > > this
> > > > >>> > endponit?
> > > > >>> >
> > > > >>> > --
> > > > >>> > Avinash Sridharan, Mesosphere
> > > > >>> > +1 (323) 702 5245
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Avinash Sridharan, Mesosphere
> > > > >> +1 (323) 702 5245
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Avinash Sridharan, Mesosphere
> > > > > +1 (323) 702 5245
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Qian Zhang <zh...@gmail.com>.
Why does "delete" need an agent restart? I think operators can just delete
the CNI network configuration file from "--network_cni_config_dir" at any
time they want, and later when a framework tries to launch a container to
that deleted CNI network, CNI isolator will find that network is in its
cache but not in the disk, so it can fail framework's request and remove
that CNI network from its cache. So it is kind of lazy delete in cache.
Thanks,
Qian Zhang
On Thu, Dec 8, 2016 at 8:12 AM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> On Wed, Dec 7, 2016 at 4:07 PM, Daniel Osborne <da...@tigera.io> wrote:
>
> > For the record, we already support a). Qian explains it here:
> > https://issues.apache.org/jira/browse/MESOS-6567?
> > focusedCommentId=15652501&page=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-15652501
> >
> > You are correct. We don't store the config in-memory just the `name`. So
> we will be reading the config every time we launch a new container. So
> looks like "delete" is the only operation that will need an agent restart.
>
> >
> > On Wed, Dec 7, 2016 at 4:02 PM, Avinash Sridharan <avinash@mesosphere.io
> >
> > wrote:
> >
> > > Thinking about the solution of treating the CNI config as an in-memory
> > > cache and doing disk reads on failures I see two problems:
> > > a) We won't be able to support modifications to CNI networks. Since
> > > modification to existing networks won't generate a miss.
> > > b) We won't be able to support deletion of CNI networks.
> > >
> > > The two operations above will still need an agent restart.
> > >
> > > On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <
> avinash@mesosphere.io
> > >
> > > wrote:
> > >
> > > >
> > > >
> > > > On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <
> > avinash@mesosphere.io
> > > >
> > > > wrote:
> > > >
> > > >>
> > > >>
> > > >> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io>
> wrote:
> > > >>
> > > >>> Chiming in since I raised an identical issue a few weeks back:
> > > >>> https://issues.apache.org/jira/browse/MESOS-6567
> > > >>>
> > > >>> The proposed endpoint solution sounds plausible. However I'd like
> to
> > > >>> explore if it solves the use case I raised my issue for. I was
> trying
> > > to
> > > >>> create a Mesos framework that adds new CNI networks. But [IIRC] the
> > > Agent
> > > >>> API can't be reached from a Mesos Executor instance since the Agent
> > > could
> > > >>> be listening on a non-default port, or on any of its IPs. The
> > executor
> > > >>> instance doesn't know that information, so after it installs the
> > > plugin,
> > > >>> it
> > > >>> won't know how to reach that new reload endpoint.
> > > >>>
> > > >>
> > > >> Just trying to understand the problem you are alluding to here. The
> > > >> executor needs to register with the agent in order to launch the
> > > container,
> > > >> so it should have reachability to the agent, and hence the endpoint?
> > > >>
> > > >>
> > > >>> - Is there a reliable way to reach the reload endpoint from a
> > default
> > > >>> executor instance?
> > > >>> - Why not scan the config directory every time? Are you trying to
> > avoid
> > > >>> the
> > > >>> speed hit from disk reads?
> > > >>>
> > > >> By scan the config directory every time, do you mean run a timer
> that
> > > >> will periodically scan the config directory and keep updating the
> > > configs.
> > > >> This is feasible. The only problem is that the point at which the
> > > operator
> > > >> write the config and the point at which the network will be
> available
> > > for
> > > >> container launch will not be deterministic. The behavior would be
> much
> > > >> cleaner if we can make it deterministic.
> > > >>
> > > >
> > > > Daniel, ignore this comment. I think you were referring to using the
> > disc
> > > > as a cache as Vinod had pointed out. I misread your suggestion.
> > > >
> > > >> Best,
> > > >>> -Dan
> > > >>>
> > > >>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <
> > > avinash@mesosphere.io
> > > >>> >
> > > >>> wrote:
> > > >>>
> > > >>> > @adam @vinod Starting to work on
> > > >>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some
> > inputs.
> > > >>> >
> > > >>> >
> > > >>> > The goal is to allow the `network/cni` isolator to load CNI
> configs
> > > >>> without
> > > >>> > the need for agent restarts. Had a discussion with @jieyu and the
> > > >>> solution
> > > >>> > we came up with was for the `network/cni` isolator to expose an
> > > >>> endpoint
> > > >>> > called `reload`. The endpoint will accept `POST` requests (with
> an
> > > >>> empty
> > > >>> > body), which will trigger the `network/cni` isolator to reload
> the
> > > CNI
> > > >>> > configs present in the `network_cni_config_dir`. On a successful
> > > >>> `reload`
> > > >>> > the `network/cni` isolator will respond with an empty HTTP
> > response.
> > > >>> Wanted
> > > >>> > to run this by you guys to understand the implications on
> > authn/authz
> > > >>> and
> > > >>> > if this is the right place (the `network/cni` isolator) to host
> > this
> > > >>> > endponit?
> > > >>> >
> > > >>> > --
> > > >>> > Avinash Sridharan, Mesosphere
> > > >>> > +1 (323) 702 5245
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Avinash Sridharan, Mesosphere
> > > >> +1 (323) 702 5245
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
> > >
> > >
> > > --
> > > Avinash Sridharan, Mesosphere
> > > +1 (323) 702 5245
> > >
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
On Wed, Dec 7, 2016 at 4:07 PM, Daniel Osborne <da...@tigera.io> wrote:
> For the record, we already support a). Qian explains it here:
> https://issues.apache.org/jira/browse/MESOS-6567?
> focusedCommentId=15652501&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15652501
>
> You are correct. We don't store the config in-memory just the `name`. So
we will be reading the config every time we launch a new container. So
looks like "delete" is the only operation that will need an agent restart.
>
> On Wed, Dec 7, 2016 at 4:02 PM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > Thinking about the solution of treating the CNI config as an in-memory
> > cache and doing disk reads on failures I see two problems:
> > a) We won't be able to support modifications to CNI networks. Since
> > modification to existing networks won't generate a miss.
> > b) We won't be able to support deletion of CNI networks.
> >
> > The two operations above will still need an agent restart.
> >
> > On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <avinash@mesosphere.io
> >
> > wrote:
> >
> > >
> > >
> > > On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <
> avinash@mesosphere.io
> > >
> > > wrote:
> > >
> > >>
> > >>
> > >> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io> wrote:
> > >>
> > >>> Chiming in since I raised an identical issue a few weeks back:
> > >>> https://issues.apache.org/jira/browse/MESOS-6567
> > >>>
> > >>> The proposed endpoint solution sounds plausible. However I'd like to
> > >>> explore if it solves the use case I raised my issue for. I was trying
> > to
> > >>> create a Mesos framework that adds new CNI networks. But [IIRC] the
> > Agent
> > >>> API can't be reached from a Mesos Executor instance since the Agent
> > could
> > >>> be listening on a non-default port, or on any of its IPs. The
> executor
> > >>> instance doesn't know that information, so after it installs the
> > plugin,
> > >>> it
> > >>> won't know how to reach that new reload endpoint.
> > >>>
> > >>
> > >> Just trying to understand the problem you are alluding to here. The
> > >> executor needs to register with the agent in order to launch the
> > container,
> > >> so it should have reachability to the agent, and hence the endpoint?
> > >>
> > >>
> > >>> - Is there a reliable way to reach the reload endpoint from a
> default
> > >>> executor instance?
> > >>> - Why not scan the config directory every time? Are you trying to
> avoid
> > >>> the
> > >>> speed hit from disk reads?
> > >>>
> > >> By scan the config directory every time, do you mean run a timer that
> > >> will periodically scan the config directory and keep updating the
> > configs.
> > >> This is feasible. The only problem is that the point at which the
> > operator
> > >> write the config and the point at which the network will be available
> > for
> > >> container launch will not be deterministic. The behavior would be much
> > >> cleaner if we can make it deterministic.
> > >>
> > >
> > > Daniel, ignore this comment. I think you were referring to using the
> disc
> > > as a cache as Vinod had pointed out. I misread your suggestion.
> > >
> > >> Best,
> > >>> -Dan
> > >>>
> > >>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <
> > avinash@mesosphere.io
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > @adam @vinod Starting to work on
> > >>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some
> inputs.
> > >>> >
> > >>> >
> > >>> > The goal is to allow the `network/cni` isolator to load CNI configs
> > >>> without
> > >>> > the need for agent restarts. Had a discussion with @jieyu and the
> > >>> solution
> > >>> > we came up with was for the `network/cni` isolator to expose an
> > >>> endpoint
> > >>> > called `reload`. The endpoint will accept `POST` requests (with an
> > >>> empty
> > >>> > body), which will trigger the `network/cni` isolator to reload the
> > CNI
> > >>> > configs present in the `network_cni_config_dir`. On a successful
> > >>> `reload`
> > >>> > the `network/cni` isolator will respond with an empty HTTP
> response.
> > >>> Wanted
> > >>> > to run this by you guys to understand the implications on
> authn/authz
> > >>> and
> > >>> > if this is the right place (the `network/cni` isolator) to host
> this
> > >>> > endponit?
> > >>> >
> > >>> > --
> > >>> > Avinash Sridharan, Mesosphere
> > >>> > +1 (323) 702 5245
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Avinash Sridharan, Mesosphere
> > >> +1 (323) 702 5245
> > >>
> > >
> > >
> > >
> > > --
> > > Avinash Sridharan, Mesosphere
> > > +1 (323) 702 5245
> > >
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Daniel Osborne <da...@tigera.io>.
For the record, we already support a). Qian explains it here:
https://issues.apache.org/jira/browse/MESOS-6567?focusedCommentId=15652501&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15652501
On Wed, Dec 7, 2016 at 4:02 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> Thinking about the solution of treating the CNI config as an in-memory
> cache and doing disk reads on failures I see two problems:
> a) We won't be able to support modifications to CNI networks. Since
> modification to existing networks won't generate a miss.
> b) We won't be able to support deletion of CNI networks.
>
> The two operations above will still need an agent restart.
>
> On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> >
> >
> > On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <avinash@mesosphere.io
> >
> > wrote:
> >
> >>
> >>
> >> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io> wrote:
> >>
> >>> Chiming in since I raised an identical issue a few weeks back:
> >>> https://issues.apache.org/jira/browse/MESOS-6567
> >>>
> >>> The proposed endpoint solution sounds plausible. However I'd like to
> >>> explore if it solves the use case I raised my issue for. I was trying
> to
> >>> create a Mesos framework that adds new CNI networks. But [IIRC] the
> Agent
> >>> API can't be reached from a Mesos Executor instance since the Agent
> could
> >>> be listening on a non-default port, or on any of its IPs. The executor
> >>> instance doesn't know that information, so after it installs the
> plugin,
> >>> it
> >>> won't know how to reach that new reload endpoint.
> >>>
> >>
> >> Just trying to understand the problem you are alluding to here. The
> >> executor needs to register with the agent in order to launch the
> container,
> >> so it should have reachability to the agent, and hence the endpoint?
> >>
> >>
> >>> - Is there a reliable way to reach the reload endpoint from a default
> >>> executor instance?
> >>> - Why not scan the config directory every time? Are you trying to avoid
> >>> the
> >>> speed hit from disk reads?
> >>>
> >> By scan the config directory every time, do you mean run a timer that
> >> will periodically scan the config directory and keep updating the
> configs.
> >> This is feasible. The only problem is that the point at which the
> operator
> >> write the config and the point at which the network will be available
> for
> >> container launch will not be deterministic. The behavior would be much
> >> cleaner if we can make it deterministic.
> >>
> >
> > Daniel, ignore this comment. I think you were referring to using the disc
> > as a cache as Vinod had pointed out. I misread your suggestion.
> >
> >> Best,
> >>> -Dan
> >>>
> >>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <
> avinash@mesosphere.io
> >>> >
> >>> wrote:
> >>>
> >>> > @adam @vinod Starting to work on
> >>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
> >>> >
> >>> >
> >>> > The goal is to allow the `network/cni` isolator to load CNI configs
> >>> without
> >>> > the need for agent restarts. Had a discussion with @jieyu and the
> >>> solution
> >>> > we came up with was for the `network/cni` isolator to expose an
> >>> endpoint
> >>> > called `reload`. The endpoint will accept `POST` requests (with an
> >>> empty
> >>> > body), which will trigger the `network/cni` isolator to reload the
> CNI
> >>> > configs present in the `network_cni_config_dir`. On a successful
> >>> `reload`
> >>> > the `network/cni` isolator will respond with an empty HTTP response.
> >>> Wanted
> >>> > to run this by you guys to understand the implications on authn/authz
> >>> and
> >>> > if this is the right place (the `network/cni` isolator) to host this
> >>> > endponit?
> >>> >
> >>> > --
> >>> > Avinash Sridharan, Mesosphere
> >>> > +1 (323) 702 5245
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Avinash Sridharan, Mesosphere
> >> +1 (323) 702 5245
> >>
> >
> >
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
Thinking about the solution of treating the CNI config as an in-memory
cache and doing disk reads on failures I see two problems:
a) We won't be able to support modifications to CNI networks. Since
modification to existing networks won't generate a miss.
b) We won't be able to support deletion of CNI networks.
The two operations above will still need an agent restart.
On Wed, Dec 7, 2016 at 3:40 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
>
>
> On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
>>
>>
>> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io> wrote:
>>
>>> Chiming in since I raised an identical issue a few weeks back:
>>> https://issues.apache.org/jira/browse/MESOS-6567
>>>
>>> The proposed endpoint solution sounds plausible. However I'd like to
>>> explore if it solves the use case I raised my issue for. I was trying to
>>> create a Mesos framework that adds new CNI networks. But [IIRC] the Agent
>>> API can't be reached from a Mesos Executor instance since the Agent could
>>> be listening on a non-default port, or on any of its IPs. The executor
>>> instance doesn't know that information, so after it installs the plugin,
>>> it
>>> won't know how to reach that new reload endpoint.
>>>
>>
>> Just trying to understand the problem you are alluding to here. The
>> executor needs to register with the agent in order to launch the container,
>> so it should have reachability to the agent, and hence the endpoint?
>>
>>
>>> - Is there a reliable way to reach the reload endpoint from a default
>>> executor instance?
>>> - Why not scan the config directory every time? Are you trying to avoid
>>> the
>>> speed hit from disk reads?
>>>
>> By scan the config directory every time, do you mean run a timer that
>> will periodically scan the config directory and keep updating the configs.
>> This is feasible. The only problem is that the point at which the operator
>> write the config and the point at which the network will be available for
>> container launch will not be deterministic. The behavior would be much
>> cleaner if we can make it deterministic.
>>
>
> Daniel, ignore this comment. I think you were referring to using the disc
> as a cache as Vinod had pointed out. I misread your suggestion.
>
>> Best,
>>> -Dan
>>>
>>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <avinash@mesosphere.io
>>> >
>>> wrote:
>>>
>>> > @adam @vinod Starting to work on
>>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
>>> >
>>> >
>>> > The goal is to allow the `network/cni` isolator to load CNI configs
>>> without
>>> > the need for agent restarts. Had a discussion with @jieyu and the
>>> solution
>>> > we came up with was for the `network/cni` isolator to expose an
>>> endpoint
>>> > called `reload`. The endpoint will accept `POST` requests (with an
>>> empty
>>> > body), which will trigger the `network/cni` isolator to reload the CNI
>>> > configs present in the `network_cni_config_dir`. On a successful
>>> `reload`
>>> > the `network/cni` isolator will respond with an empty HTTP response.
>>> Wanted
>>> > to run this by you guys to understand the implications on authn/authz
>>> and
>>> > if this is the right place (the `network/cni` isolator) to host this
>>> > endponit?
>>> >
>>> > --
>>> > Avinash Sridharan, Mesosphere
>>> > +1 (323) 702 5245
>>> >
>>>
>>
>>
>>
>> --
>> Avinash Sridharan, Mesosphere
>> +1 (323) 702 5245
>>
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
On Wed, Dec 7, 2016 at 3:31 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
>
>
> On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io> wrote:
>
>> Chiming in since I raised an identical issue a few weeks back:
>> https://issues.apache.org/jira/browse/MESOS-6567
>>
>> The proposed endpoint solution sounds plausible. However I'd like to
>> explore if it solves the use case I raised my issue for. I was trying to
>> create a Mesos framework that adds new CNI networks. But [IIRC] the Agent
>> API can't be reached from a Mesos Executor instance since the Agent could
>> be listening on a non-default port, or on any of its IPs. The executor
>> instance doesn't know that information, so after it installs the plugin,
>> it
>> won't know how to reach that new reload endpoint.
>>
>
> Just trying to understand the problem you are alluding to here. The
> executor needs to register with the agent in order to launch the container,
> so it should have reachability to the agent, and hence the endpoint?
>
>
>> - Is there a reliable way to reach the reload endpoint from a default
>> executor instance?
>> - Why not scan the config directory every time? Are you trying to avoid
>> the
>> speed hit from disk reads?
>>
> By scan the config directory every time, do you mean run a timer that will
> periodically scan the config directory and keep updating the configs. This
> is feasible. The only problem is that the point at which the operator write
> the config and the point at which the network will be available for
> container launch will not be deterministic. The behavior would be much
> cleaner if we can make it deterministic.
>
Daniel, ignore this comment. I think you were referring to using the disc
as a cache as Vinod had pointed out. I misread your suggestion.
> Best,
>> -Dan
>>
>> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <av...@mesosphere.io>
>> wrote:
>>
>> > @adam @vinod Starting to work on
>> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
>> >
>> >
>> > The goal is to allow the `network/cni` isolator to load CNI configs
>> without
>> > the need for agent restarts. Had a discussion with @jieyu and the
>> solution
>> > we came up with was for the `network/cni` isolator to expose an endpoint
>> > called `reload`. The endpoint will accept `POST` requests (with an empty
>> > body), which will trigger the `network/cni` isolator to reload the CNI
>> > configs present in the `network_cni_config_dir`. On a successful
>> `reload`
>> > the `network/cni` isolator will respond with an empty HTTP response.
>> Wanted
>> > to run this by you guys to understand the implications on authn/authz
>> and
>> > if this is the right place (the `network/cni` isolator) to host this
>> > endponit?
>> >
>> > --
>> > Avinash Sridharan, Mesosphere
>> > +1 (323) 702 5245
>> >
>>
>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Avinash Sridharan <av...@mesosphere.io>.
On Wed, Dec 7, 2016 at 3:17 PM, Daniel Osborne <da...@tigera.io> wrote:
> Chiming in since I raised an identical issue a few weeks back:
> https://issues.apache.org/jira/browse/MESOS-6567
>
> The proposed endpoint solution sounds plausible. However I'd like to
> explore if it solves the use case I raised my issue for. I was trying to
> create a Mesos framework that adds new CNI networks. But [IIRC] the Agent
> API can't be reached from a Mesos Executor instance since the Agent could
> be listening on a non-default port, or on any of its IPs. The executor
> instance doesn't know that information, so after it installs the plugin, it
> won't know how to reach that new reload endpoint.
>
Just trying to understand the problem you are alluding to here. The
executor needs to register with the agent in order to launch the container,
so it should have reachability to the agent, and hence the endpoint?
> - Is there a reliable way to reach the reload endpoint from a default
> executor instance?
> - Why not scan the config directory every time? Are you trying to avoid the
> speed hit from disk reads?
>
By scan the config directory every time, do you mean run a timer that will
periodically scan the config directory and keep updating the configs. This
is feasible. The only problem is that the point at which the operator write
the config and the point at which the network will be available for
container launch will not be deterministic. The behavior would be much
cleaner if we can make it deterministic.
>
> Best,
> -Dan
>
> On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <av...@mesosphere.io>
> wrote:
>
> > @adam @vinod Starting to work on
> > https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
> >
> >
> > The goal is to allow the `network/cni` isolator to load CNI configs
> without
> > the need for agent restarts. Had a discussion with @jieyu and the
> solution
> > we came up with was for the `network/cni` isolator to expose an endpoint
> > called `reload`. The endpoint will accept `POST` requests (with an empty
> > body), which will trigger the `network/cni` isolator to reload the CNI
> > configs present in the `network_cni_config_dir`. On a successful `reload`
> > the `network/cni` isolator will respond with an empty HTTP response.
> Wanted
> > to run this by you guys to understand the implications on authn/authz and
> > if this is the right place (the `network/cni` isolator) to host this
> > endponit?
> >
> > --
> > Avinash Sridharan, Mesosphere
> > +1 (323) 702 5245
> >
>
--
Avinash Sridharan, Mesosphere
+1 (323) 702 5245
Re: Adding a reload end-point to `network/cni` isolator
Posted by Daniel Osborne <da...@tigera.io>.
Chiming in since I raised an identical issue a few weeks back:
https://issues.apache.org/jira/browse/MESOS-6567
The proposed endpoint solution sounds plausible. However I'd like to
explore if it solves the use case I raised my issue for. I was trying to
create a Mesos framework that adds new CNI networks. But [IIRC] the Agent
API can't be reached from a Mesos Executor instance since the Agent could
be listening on a non-default port, or on any of its IPs. The executor
instance doesn't know that information, so after it installs the plugin, it
won't know how to reach that new reload endpoint.
- Is there a reliable way to reach the reload endpoint from a default
executor instance?
- Why not scan the config directory every time? Are you trying to avoid the
speed hit from disk reads?
Best,
-Dan
On Wed, Dec 7, 2016 at 3:01 PM, Avinash Sridharan <av...@mesosphere.io>
wrote:
> @adam @vinod Starting to work on
> https://issues.apache.org/jira/browse/MESOS-6679 . Need some inputs.
>
>
> The goal is to allow the `network/cni` isolator to load CNI configs without
> the need for agent restarts. Had a discussion with @jieyu and the solution
> we came up with was for the `network/cni` isolator to expose an endpoint
> called `reload`. The endpoint will accept `POST` requests (with an empty
> body), which will trigger the `network/cni` isolator to reload the CNI
> configs present in the `network_cni_config_dir`. On a successful `reload`
> the `network/cni` isolator will respond with an empty HTTP response. Wanted
> to run this by you guys to understand the implications on authn/authz and
> if this is the right place (the `network/cni` isolator) to host this
> endponit?
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>