You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Shuai Lin <li...@gmail.com> on 2015/12/01 15:12:22 UTC
Re: Injecting data into mesos container
On Tue, Dec 1, 2015 at 1:28 AM, Adam Medziński <ad...@gmail.com>
wrote:
> My fault, my description of the problem was too general. I need to
> pass confidential data to Mesos task in a secure manner. By that data
> I mean credentials (private keys, password etc.) to other company
> systems that are generated for a particular task on Mesos. To run
> tasks we use Marathon framework. By Mesos Containerizer I mean
> https://mesos.apache.org/documentation/latest/containerizer/. From the
> previous statements it seems to me that the best solution would be to
> write hook that will generate credentials and set them as environment
> variables.
>
Since the web ui is on port 5050, which is usually blocked from outside by
firewall rules. If one have access to your mesos web ui, it's very likely
that he can access the environment variable of tasks info as well, e.g.
from mesos master/slave http api or marathon api.
> On 30 November 2015 at 17:22, tommy xiao <xi...@gmail.com> wrote:
> > Hi Adam,
> >
> > could you please give more details on your env. what your mean for mesos
> > container? it point to Docker? or another mesos containerizer. if my
> > remember correctly, the mesos containerizer is native sandbox through
> > cgroup. you can use URI to fetch files on the sandbox, it match your
> > requirements. but you said it not yet match your require, what your real
> > requirement?
> >
> > 2015-11-30 23:32 GMT+08:00 Adam Medziński <ad...@gmail.com>:
> >
> >> Yes, but it will also make that data available in mesos web UI. I need
> >> this data to be available only for task process.
> >>
> >> On 30 November 2015 at 16:20, Shuai Lin <li...@gmail.com> wrote:
> >> > Also mesos supports specifying a list of uris in the command info,
> which
> >> > you can manipulate in your framework, and mesos would fetch files from
> >> > those uris into the task's sandbox.
> >> >
> >> >
> >>
> https://github.com/apache/mesos/blob/b2b0eed/include/mesos/mesos.proto#L391
> >> .
> >> >
> >> >
> >> > On Mon, Nov 30, 2015 at 7:53 PM, tommy xiao <xi...@gmail.com> wrote:
> >> >
> >> >> if the external volume is valid, we can use hdfs as alternative
> service
> >> to
> >> >> provide the storage solution.
> >> >>
> >> >> 2015-11-30 19:34 GMT+08:00 Vaibhav Khanduja <
> vaibhavkhanduja@gmail.com
> >> >:
> >> >>
> >> >> > One possible way could be map an external volume and make the
> >> certificate
> >> >> > available on the volume. The application can poll the file, and
> >> presence
> >> >> of
> >> >> > file can trigger alert to read it.
> >> >> > 2cnts
> >> >> >
> >> >> > On Mon, Nov 30, 2015 at 4:56 PM, tommy xiao <xi...@gmail.com>
> wrote:
> >> >> >
> >> >> > > Haosdent's suggest is not ideal way.
> >> >> > >
> >> >> > > 2015-11-30 0:28 GMT+08:00 Jojy Varghese <jo...@mesosphere.io>:
> >> >> > >
> >> >> > > > Haosdent is right that “nsenter” is used by docker users but
> >> wanted
> >> >> to
> >> >> > > add
> >> >> > > > that it is not a docker only command. It is a linux utility
> that
> >> can
> >> >> be
> >> >> > > > used to enter the namespaces of processes using their “procfs”
> >> files
> >> >> > [1].
> >> >> > > > Mesos containerizer can be configured to use linux isolators
> >> >> > (filesystem
> >> >> > > > for example) which uses namespaces. So these namespaces (say
> mount
> >> >> > > > namespace) can be then entered using “nsenter”.
> >> >> > > > If i understand what you are trying to achieve, the
> certificates
> >> can
> >> >> > be
> >> >> > > > placed into the container by entering the mount namespace of
> the
> >> >> > > container
> >> >> > > > process. Mount namespace will give you view of the file system
> as
> >> >> > viewed
> >> >> > > by
> >> >> > > > the container process.
> >> >> > > >
> >> >> > > > -Jojy
> >> >> > > >
> >> >> > > > [1] http://man7.org/linux/man-pages/man1/nsenter.1.html <
> >> >> > > > http://man7.org/linux/man-pages/man1/nsenter.1.html>
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > > On Nov 29, 2015, at 4:51 AM, haosdent <ha...@gmail.com>
> >> wrote:
> >> >> > > > >
> >> >> > > > > If you use Docker container, you could use "nsenter" command
> >> >> > > > > https://docs.docker.com/engine/reference/commandline/exec/ .
> >> But
> >> >> > > because
> >> >> > > > > you use Mesos container, I think in your task, you need send
> the
> >> >> data
> >> >> > > to
> >> >> > > > > some store, maybe just print data to task logs or save them
> in
> >> >> > > database.
> >> >> > > > So
> >> >> > > > > that we could inspect them outside Mesos container.
> >> >> > > > >
> >> >> > > > > On Sun, Nov 29, 2015 at 8:39 PM, Shuai Lin <
> >> linshuai2012@gmail.com
> >> >> >
> >> >> > > > wrote:
> >> >> > > > >
> >> >> > > > >> This ticket may be related:
> >> >> > > > >> https://issues.apache.org/jira/browse/MESOS-2724
> >> >> > > > >> "Support running custom commands on slaves when launching a
> >> docker
> >> >> > > > >> container"
> >> >> > > > >>
> >> >> > > > >> On Sun, Nov 29, 2015 at 6:13 PM, Adam Medziński <
> >> >> > > > adam.medzinski@gmail.com>
> >> >> > > > >> wrote:
> >> >> > > > >>
> >> >> > > > >>> So if I understand all correctly - I can (as Sargun Dhillon
> >> >> > suggests)
> >> >> > > > >>> write my own mesos hook and inject data through environment
> >> >> > > variables,
> >> >> > > > >>> or try to use nsenter if I need something more
> sophisticated?
> >> >> > > > >>>
> >> >> > > > >>> On 28 November 2015 at 17:23, Jojy Varghese <
> >> jojy@mesosphere.io>
> >> >> > > > wrote:
> >> >> > > > >>>> Although its not possible today directly using Mesos, we
> >> could
> >> >> > > always
> >> >> > > > >>> use “nsenter” to enter the container namespace and do
> things.
> >> I
> >> >> > > haven’t
> >> >> > > > >>> tried it myself but conceptually thats the way to do it.
> >> >> > > > >>>>
> >> >> > > > >>>> -Jojy
> >> >> > > > >>>>
> >> >> > > > >>>>
> >> >> > > > >>>>> On Nov 28, 2015, at 3:30 AM, Sargun Dhillon <
> >> sargun@sargun.me>
> >> >> > > > wrote:
> >> >> > > > >>>>>
> >> >> > > > >>>>> You can do this using Mesos Modules - Documented here:
> >> >> > > > >>>>> http://mesos.apache.org/documentation/latest/modules/
> >> >> > > > >>>>>
> >> >> > > > >>>>> I think you're probably looking at writing a hooks module
> >> for
> >> >> the
> >> >> > > > >>>>> agent, and taking advantage of the
> >> >> > > slaveExecutorEnvironmentDecorator
> >> >> > > > >>>>> callback.
> >> >> > > > >>>>>
> >> >> > > > >>>>> On Sat, Nov 28, 2015 at 3:12 AM, tommy xiao <
> >> xiaods@gmail.com>
> >> >> > > > wrote:
> >> >> > > > >>>>>> feel no way, you need some tricks on it.
> >> >> > > > >>>>>>
> >> >> > > > >>>>>> 2015-11-27 19:23 GMT+08:00 Adam Medziński <
> >> >> > > adam.medzinski@gmail.com
> >> >> > > > >>> :
> >> >> > > > >>>>>>
> >> >> > > > >>>>>>> It is possible to inject dynamically generated data
> (for
> >> >> > example
> >> >> > > > >>>>>>> certificate generated for task) into mesos container?
> Data
> >> >> > should
> >> >> > > > be
> >> >> > > > >>>>>>> available only to task process.
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> --
> >> >> > > > >>>>>>> Best Regards,
> >> >> > > > >>>>>>> Adam Medziński
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>
> >> >> > > > >>>>>>
> >> >> > > > >>>>>>
> >> >> > > > >>>>>> --
> >> >> > > > >>>>>> Deshi Xiao
> >> >> > > > >>>>>> Twitter: xds2000
> >> >> > > > >>>>>> E-mail: xiaods(AT)gmail.com
> >> >> > > > >>>>
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>> --
> >> >> > > > >>> Best Regards,
> >> >> > > > >>> Adam Medziński
> >> >> > > > >>>
> >> >> > > > >>
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > --
> >> >> > > > > Best Regards,
> >> >> > > > > Haosdent Huang
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > Deshi Xiao
> >> >> > > Twitter: xds2000
> >> >> > > E-mail: xiaods(AT)gmail.com
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Deshi Xiao
> >> >> Twitter: xds2000
> >> >> E-mail: xiaods(AT)gmail.com
> >> >>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Adam Medziński
> >>
> >
> >
> >
> > --
> > Deshi Xiao
> > Twitter: xds2000
> > E-mail: xiaods(AT)gmail.com
>
>
>
> --
> Best Regards,
> Adam Medziński
>
Re: Injecting data into mesos container
Posted by Shuai Lin <li...@gmail.com>.
Ah, I just checked the /state.json endpoint on slave and found it won't
return the environment variables of tasks in the response, though in the
source code I found it could be included:
https://github.com/apache/mesos/blob/0.26.0-rc2/src/common/http.cpp#L288-L300
So I still suggest you go through all the possible http api endpoint to
make sure of it.
On Tue, Dec 1, 2015 at 10:12 PM, Shuai Lin <li...@gmail.com> wrote:
>
> On Tue, Dec 1, 2015 at 1:28 AM, Adam Medziński <ad...@gmail.com>
> wrote:
>
>> My fault, my description of the problem was too general. I need to
>> pass confidential data to Mesos task in a secure manner. By that data
>> I mean credentials (private keys, password etc.) to other company
>> systems that are generated for a particular task on Mesos. To run
>> tasks we use Marathon framework. By Mesos Containerizer I mean
>> https://mesos.apache.org/documentation/latest/containerizer/. From the
>> previous statements it seems to me that the best solution would be to
>> write hook that will generate credentials and set them as environment
>> variables.
>>
>
> Since the web ui is on port 5050, which is usually blocked from outside by
> firewall rules. If one have access to your mesos web ui, it's very likely
> that he can access the environment variable of tasks info as well, e.g.
> from mesos master/slave http api or marathon api.
>
>
>> On 30 November 2015 at 17:22, tommy xiao <xi...@gmail.com> wrote:
>> > Hi Adam,
>> >
>> > could you please give more details on your env. what your mean for mesos
>> > container? it point to Docker? or another mesos containerizer. if my
>> > remember correctly, the mesos containerizer is native sandbox through
>> > cgroup. you can use URI to fetch files on the sandbox, it match your
>> > requirements. but you said it not yet match your require, what your real
>> > requirement?
>> >
>> > 2015-11-30 23:32 GMT+08:00 Adam Medziński <ad...@gmail.com>:
>> >
>> >> Yes, but it will also make that data available in mesos web UI. I need
>> >> this data to be available only for task process.
>> >>
>> >> On 30 November 2015 at 16:20, Shuai Lin <li...@gmail.com>
>> wrote:
>> >> > Also mesos supports specifying a list of uris in the command info,
>> which
>> >> > you can manipulate in your framework, and mesos would fetch files
>> from
>> >> > those uris into the task's sandbox.
>> >> >
>> >> >
>> >>
>> https://github.com/apache/mesos/blob/b2b0eed/include/mesos/mesos.proto#L391
>> >> .
>> >> >
>> >> >
>> >> > On Mon, Nov 30, 2015 at 7:53 PM, tommy xiao <xi...@gmail.com>
>> wrote:
>> >> >
>> >> >> if the external volume is valid, we can use hdfs as alternative
>> service
>> >> to
>> >> >> provide the storage solution.
>> >> >>
>> >> >> 2015-11-30 19:34 GMT+08:00 Vaibhav Khanduja <
>> vaibhavkhanduja@gmail.com
>> >> >:
>> >> >>
>> >> >> > One possible way could be map an external volume and make the
>> >> certificate
>> >> >> > available on the volume. The application can poll the file, and
>> >> presence
>> >> >> of
>> >> >> > file can trigger alert to read it.
>> >> >> > 2cnts
>> >> >> >
>> >> >> > On Mon, Nov 30, 2015 at 4:56 PM, tommy xiao <xi...@gmail.com>
>> wrote:
>> >> >> >
>> >> >> > > Haosdent's suggest is not ideal way.
>> >> >> > >
>> >> >> > > 2015-11-30 0:28 GMT+08:00 Jojy Varghese <jo...@mesosphere.io>:
>> >> >> > >
>> >> >> > > > Haosdent is right that “nsenter” is used by docker users but
>> >> wanted
>> >> >> to
>> >> >> > > add
>> >> >> > > > that it is not a docker only command. It is a linux utility
>> that
>> >> can
>> >> >> be
>> >> >> > > > used to enter the namespaces of processes using their “procfs”
>> >> files
>> >> >> > [1].
>> >> >> > > > Mesos containerizer can be configured to use linux isolators
>> >> >> > (filesystem
>> >> >> > > > for example) which uses namespaces. So these namespaces (say
>> mount
>> >> >> > > > namespace) can be then entered using “nsenter”.
>> >> >> > > > If i understand what you are trying to achieve, the
>> certificates
>> >> can
>> >> >> > be
>> >> >> > > > placed into the container by entering the mount namespace of
>> the
>> >> >> > > container
>> >> >> > > > process. Mount namespace will give you view of the file
>> system as
>> >> >> > viewed
>> >> >> > > by
>> >> >> > > > the container process.
>> >> >> > > >
>> >> >> > > > -Jojy
>> >> >> > > >
>> >> >> > > > [1] http://man7.org/linux/man-pages/man1/nsenter.1.html <
>> >> >> > > > http://man7.org/linux/man-pages/man1/nsenter.1.html>
>> >> >> > > >
>> >> >> > > >
>> >> >> > > >
>> >> >> > > > > On Nov 29, 2015, at 4:51 AM, haosdent <ha...@gmail.com>
>> >> wrote:
>> >> >> > > > >
>> >> >> > > > > If you use Docker container, you could use "nsenter" command
>> >> >> > > > > https://docs.docker.com/engine/reference/commandline/exec/
>> .
>> >> But
>> >> >> > > because
>> >> >> > > > > you use Mesos container, I think in your task, you need
>> send the
>> >> >> data
>> >> >> > > to
>> >> >> > > > > some store, maybe just print data to task logs or save them
>> in
>> >> >> > > database.
>> >> >> > > > So
>> >> >> > > > > that we could inspect them outside Mesos container.
>> >> >> > > > >
>> >> >> > > > > On Sun, Nov 29, 2015 at 8:39 PM, Shuai Lin <
>> >> linshuai2012@gmail.com
>> >> >> >
>> >> >> > > > wrote:
>> >> >> > > > >
>> >> >> > > > >> This ticket may be related:
>> >> >> > > > >> https://issues.apache.org/jira/browse/MESOS-2724
>> >> >> > > > >> "Support running custom commands on slaves when launching a
>> >> docker
>> >> >> > > > >> container"
>> >> >> > > > >>
>> >> >> > > > >> On Sun, Nov 29, 2015 at 6:13 PM, Adam Medziński <
>> >> >> > > > adam.medzinski@gmail.com>
>> >> >> > > > >> wrote:
>> >> >> > > > >>
>> >> >> > > > >>> So if I understand all correctly - I can (as Sargun
>> Dhillon
>> >> >> > suggests)
>> >> >> > > > >>> write my own mesos hook and inject data through
>> environment
>> >> >> > > variables,
>> >> >> > > > >>> or try to use nsenter if I need something more
>> sophisticated?
>> >> >> > > > >>>
>> >> >> > > > >>> On 28 November 2015 at 17:23, Jojy Varghese <
>> >> jojy@mesosphere.io>
>> >> >> > > > wrote:
>> >> >> > > > >>>> Although its not possible today directly using Mesos, we
>> >> could
>> >> >> > > always
>> >> >> > > > >>> use “nsenter” to enter the container namespace and do
>> things.
>> >> I
>> >> >> > > haven’t
>> >> >> > > > >>> tried it myself but conceptually thats the way to do it.
>> >> >> > > > >>>>
>> >> >> > > > >>>> -Jojy
>> >> >> > > > >>>>
>> >> >> > > > >>>>
>> >> >> > > > >>>>> On Nov 28, 2015, at 3:30 AM, Sargun Dhillon <
>> >> sargun@sargun.me>
>> >> >> > > > wrote:
>> >> >> > > > >>>>>
>> >> >> > > > >>>>> You can do this using Mesos Modules - Documented here:
>> >> >> > > > >>>>> http://mesos.apache.org/documentation/latest/modules/
>> >> >> > > > >>>>>
>> >> >> > > > >>>>> I think you're probably looking at writing a hooks
>> module
>> >> for
>> >> >> the
>> >> >> > > > >>>>> agent, and taking advantage of the
>> >> >> > > slaveExecutorEnvironmentDecorator
>> >> >> > > > >>>>> callback.
>> >> >> > > > >>>>>
>> >> >> > > > >>>>> On Sat, Nov 28, 2015 at 3:12 AM, tommy xiao <
>> >> xiaods@gmail.com>
>> >> >> > > > wrote:
>> >> >> > > > >>>>>> feel no way, you need some tricks on it.
>> >> >> > > > >>>>>>
>> >> >> > > > >>>>>> 2015-11-27 19:23 GMT+08:00 Adam Medziński <
>> >> >> > > adam.medzinski@gmail.com
>> >> >> > > > >>> :
>> >> >> > > > >>>>>>
>> >> >> > > > >>>>>>> It is possible to inject dynamically generated data
>> (for
>> >> >> > example
>> >> >> > > > >>>>>>> certificate generated for task) into mesos container?
>> Data
>> >> >> > should
>> >> >> > > > be
>> >> >> > > > >>>>>>> available only to task process.
>> >> >> > > > >>>>>>>
>> >> >> > > > >>>>>>> --
>> >> >> > > > >>>>>>> Best Regards,
>> >> >> > > > >>>>>>> Adam Medziński
>> >> >> > > > >>>>>>>
>> >> >> > > > >>>>>>
>> >> >> > > > >>>>>>
>> >> >> > > > >>>>>>
>> >> >> > > > >>>>>> --
>> >> >> > > > >>>>>> Deshi Xiao
>> >> >> > > > >>>>>> Twitter: xds2000
>> >> >> > > > >>>>>> E-mail: xiaods(AT)gmail.com
>> >> >> > > > >>>>
>> >> >> > > > >>>
>> >> >> > > > >>>
>> >> >> > > > >>>
>> >> >> > > > >>> --
>> >> >> > > > >>> Best Regards,
>> >> >> > > > >>> Adam Medziński
>> >> >> > > > >>>
>> >> >> > > > >>
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > > --
>> >> >> > > > > Best Regards,
>> >> >> > > > > Haosdent Huang
>> >> >> > > >
>> >> >> > > >
>> >> >> > >
>> >> >> > >
>> >> >> > > --
>> >> >> > > Deshi Xiao
>> >> >> > > Twitter: xds2000
>> >> >> > > E-mail: xiaods(AT)gmail.com
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Deshi Xiao
>> >> >> Twitter: xds2000
>> >> >> E-mail: xiaods(AT)gmail.com
>> >> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Adam Medziński
>> >>
>> >
>> >
>> >
>> > --
>> > Deshi Xiao
>> > Twitter: xds2000
>> > E-mail: xiaods(AT)gmail.com
>>
>>
>>
>> --
>> Best Regards,
>> Adam Medziński
>>
>
>