You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Jie Yu <yu...@gmail.com> on 2015/11/11 00:45:03 UTC
Fetcher refactor proposal
Hi,
Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
binary) for executors/tasks. A recent refactor (MESOS-336
<https://issues.apache.org/jira/browse/MESOS-336>) added caching support to
the fetcher. The recent work on filesystem isolation/unified containerizer (
MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
natural question is: can we leverage the fetcher to fetch those filesystem
images (and cache them accordingly)? Unfortunately, the existing fetcher
interface is tightly coupled with CommandInfo::URIs for executors/tasks,
making it very hard to be re-used to fetch/cache filesystem images.
Another motivation for the refactor is that we want to extend the fetcher
to support more types of schemes. For instance, we want to support magnet
URI to enable p2p fetching. This is in fact quite important for operating a
large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>).
The goal here is to allow fetcher to be extended (e.g., using modules) so
that operators can add custom fetching support.
I proposed a solution in this doc
<https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>.
The main idea is to decouple artifacts fetching from artifacts cache
management. We can make artifacts fetching extensible (e.g. to support p2p
fetching), and solve the cache management part later.
Let me know your thoughts! Thanks!
- Jie
Re: Fetcher refactor proposal
Posted by Jie Yu <yu...@gmail.com>.
Thanks guys!
Created an Epic (MESOS-3918
<https://issues.apache.org/jira/browse/MESOS-3918>) to track.
On Wed, Nov 11, 2015 at 2:31 AM, Bernd Mathiske <be...@mesosphere.io> wrote:
> +1 - go for it!
>
> > On Nov 11, 2015, at 12:45 AM, Jie Yu <yu...@gmail.com> wrote:
> >
> > Hi,
> >
> > Fetcher was originally designed to fetch CommandInfo::URIs (e.g.,
> executor
> > binary) for executors/tasks. A recent refactor (MESOS-336
> > <https://issues.apache.org/jira/browse/MESOS-336>) added caching
> support to
> > the fetcher. The recent work on filesystem isolation/unified
> containerizer (
> > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> > natural question is: can we leverage the fetcher to fetch those
> filesystem
> > images (and cache them accordingly)? Unfortunately, the existing fetcher
> > interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> > making it very hard to be re-used to fetch/cache filesystem images.
> >
> > Another motivation for the refactor is that we want to extend the fetcher
> > to support more types of schemes. For instance, we want to support magnet
> > URI to enable p2p fetching. This is in fact quite important for
> operating a
> > large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> > The goal here is to allow fetcher to be extended (e.g., using modules) so
> > that operators can add custom fetching support.
> >
> > I proposed a solution in this doc
> > <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> >.
> > The main idea is to decouple artifacts fetching from artifacts cache
> > management. We can make artifacts fetching extensible (e.g. to support
> p2p
> > fetching), and solve the cache management part later.
> >
> > Let me know your thoughts! Thanks!
> >
> > - Jie
>
>
Re: Fetcher refactor proposal
Posted by Bernd Mathiske <be...@mesosphere.io>.
+1 - go for it!
> On Nov 11, 2015, at 12:45 AM, Jie Yu <yu...@gmail.com> wrote:
>
> Hi,
>
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <https://issues.apache.org/jira/browse/MESOS-336>) added caching support to
> the fetcher. The recent work on filesystem isolation/unified containerizer (
> MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
>
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
>
> I proposed a solution in this doc
> <https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
>
> Let me know your thoughts! Thanks!
>
> - Jie
Re: Fetcher refactor proposal
Posted by Jörg Schad <jo...@mesosphere.io>.
+1
On Wed, Nov 11, 2015 at 1:57 AM, Jie Yu <yu...@gmail.com> wrote:
> Tom, I don't have immediate plan to replace CommandInfo::URI (with
> executable/extract bits) with URI (in the doc), at least for now. The
> existing Fetcher will still perform extraction, chown, etc. for now
> (eventually, though I think those logics should be moved to
> slave/containerizer). The existing Fetcher can download the artifacts by
> leveraging the new uri::Fetcher interface (need a little refactor).
>
> On Tue, Nov 10, 2015 at 4:44 PM, Tom Arnfeld <to...@duedil.com> wrote:
>
> > This looks like a great change, btw!
> >
> > I have a quick question, how does this change affect things like the
> > executable/extract bits that are available in the existing fetcher? Would
> > that logic move outside of the fetcher itself, or would it live on the
> URI?
> >
> > I’m not sure if I’ve missed something in the design doc about this, but
> it
> > came to mind…
> >
> > Tom.
> >
> > > On 10 Nov 2015, at 23:45, Jie Yu <yu...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > Fetcher was originally designed to fetch CommandInfo::URIs (e.g.,
> > executor
> > > binary) for executors/tasks. A recent refactor (MESOS-336
> > > <https://issues.apache.org/jira/browse/MESOS-336>) added caching
> > support to
> > > the fetcher. The recent work on filesystem isolation/unified
> > containerizer (
> > > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>)
> requires
> > > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well.
> The
> > > natural question is: can we leverage the fetcher to fetch those
> > filesystem
> > > images (and cache them accordingly)? Unfortunately, the existing
> fetcher
> > > interface is tightly coupled with CommandInfo::URIs for
> executors/tasks,
> > > making it very hard to be re-used to fetch/cache filesystem images.
> > >
> > > Another motivation for the refactor is that we want to extend the
> fetcher
> > > to support more types of schemes. For instance, we want to support
> magnet
> > > URI to enable p2p fetching. This is in fact quite important for
> > operating a
> > > large cluster (MESOS-3596 <
> > https://issues.apache.org/jira/browse/MESOS-3596>).
> > > The goal here is to allow fetcher to be extended (e.g., using modules)
> so
> > > that operators can add custom fetching support.
> > >
> > > I proposed a solution in this doc
> > > <
> >
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> > >.
> > > The main idea is to decouple artifacts fetching from artifacts cache
> > > management. We can make artifacts fetching extensible (e.g. to support
> > p2p
> > > fetching), and solve the cache management part later.
> > >
> > > Let me know your thoughts! Thanks!
> > >
> > > - Jie
> >
> >
>
Re: Fetcher refactor proposal
Posted by Jie Yu <yu...@gmail.com>.
Tom, I don't have immediate plan to replace CommandInfo::URI (with
executable/extract bits) with URI (in the doc), at least for now. The
existing Fetcher will still perform extraction, chown, etc. for now
(eventually, though I think those logics should be moved to
slave/containerizer). The existing Fetcher can download the artifacts by
leveraging the new uri::Fetcher interface (need a little refactor).
On Tue, Nov 10, 2015 at 4:44 PM, Tom Arnfeld <to...@duedil.com> wrote:
> This looks like a great change, btw!
>
> I have a quick question, how does this change affect things like the
> executable/extract bits that are available in the existing fetcher? Would
> that logic move outside of the fetcher itself, or would it live on the URI?
>
> I’m not sure if I’ve missed something in the design doc about this, but it
> came to mind…
>
> Tom.
>
> > On 10 Nov 2015, at 23:45, Jie Yu <yu...@gmail.com> wrote:
> >
> > Hi,
> >
> > Fetcher was originally designed to fetch CommandInfo::URIs (e.g.,
> executor
> > binary) for executors/tasks. A recent refactor (MESOS-336
> > <https://issues.apache.org/jira/browse/MESOS-336>) added caching
> support to
> > the fetcher. The recent work on filesystem isolation/unified
> containerizer (
> > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> > natural question is: can we leverage the fetcher to fetch those
> filesystem
> > images (and cache them accordingly)? Unfortunately, the existing fetcher
> > interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> > making it very hard to be re-used to fetch/cache filesystem images.
> >
> > Another motivation for the refactor is that we want to extend the fetcher
> > to support more types of schemes. For instance, we want to support magnet
> > URI to enable p2p fetching. This is in fact quite important for
> operating a
> > large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> > The goal here is to allow fetcher to be extended (e.g., using modules) so
> > that operators can add custom fetching support.
> >
> > I proposed a solution in this doc
> > <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> >.
> > The main idea is to decouple artifacts fetching from artifacts cache
> > management. We can make artifacts fetching extensible (e.g. to support
> p2p
> > fetching), and solve the cache management part later.
> >
> > Let me know your thoughts! Thanks!
> >
> > - Jie
>
>
Re: Fetcher refactor proposal
Posted by Tom Arnfeld <to...@duedil.com>.
This looks like a great change, btw!
I have a quick question, how does this change affect things like the executable/extract bits that are available in the existing fetcher? Would that logic move outside of the fetcher itself, or would it live on the URI?
I’m not sure if I’ve missed something in the design doc about this, but it came to mind…
Tom.
> On 10 Nov 2015, at 23:45, Jie Yu <yu...@gmail.com> wrote:
>
> Hi,
>
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <https://issues.apache.org/jira/browse/MESOS-336>) added caching support to
> the fetcher. The recent work on filesystem isolation/unified containerizer (
> MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
>
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
>
> I proposed a solution in this doc
> <https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
>
> Let me know your thoughts! Thanks!
>
> - Jie
Re: Fetcher refactor proposal
Posted by Alexander Rojas <al...@mesosphere.io>.
+1
> On 11 Nov 2015, at 00:45, Jie Yu <yu...@gmail.com> wrote:
>
> Hi,
>
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <https://issues.apache.org/jira/browse/MESOS-336>) added caching support to
> the fetcher. The recent work on filesystem isolation/unified containerizer (
> MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
>
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
>
> I proposed a solution in this doc
> <https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
>
> Let me know your thoughts! Thanks!
>
> - Jie
Re: Fetcher refactor proposal
Posted by Gilbert Song <gi...@mesosphere.io>.
+1.
Depending on one of unified container issue MESOS-2970
<https://issues.apache.org/jira/browse/MESOS-2970>, We need
to support cache for container image as our next step. It would be perfect
if we can reuse some parts of fetcher cache. It may also be helpful to
many other issues related to caching.
Thank you for writing up the proposal above!
Gilbert
On Tue, Nov 10, 2015 at 3:45 PM, Jie Yu <yu...@gmail.com> wrote:
> Hi,
>
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <https://issues.apache.org/jira/browse/MESOS-336>) added caching support
> to
> the fetcher. The recent work on filesystem isolation/unified containerizer
> (
> MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
>
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
>
> I proposed a solution in this doc
> <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> >.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
>
> Let me know your thoughts! Thanks!
>
> - Jie
>