You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by olivier sallou <ol...@irisa.fr> on 2020/06/08 08:13:38 UTC
Fail to start agent 1.9.0: Failed to create memfd error
Hi,
I tried to install mesos 1.9.0 (installed from rpm) on CentOS 7. I
configured it for docker with isolation flags but it fails to start
with "Failed to open memfd file: Failed to create memfd: Invalid
argument" error (startup logs below)
Jun 8 08:03:41 mesos systemd: Started Mesos Agent.
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.938788 7002
logging.cpp:201] INFO level logging started!
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939116 7002
main.cpp:350] Build: 2019-10-22 13:58:29 by
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939124 7002
main.cpp:351] Version: 1.9.0
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939127 7002
main.cpp:354] Git tag: 1.9.0-rc3
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939131 7002
main.cpp:358] Git SHA: 5e79a584e6ec3e9e2f96e8bf418411df9dafac2e
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946902 7002
systemd.cpp:240] systemd version `219` detected
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946928 7002
main.cpp:453] Initializing systemd state
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.952776 7002
systemd.cpp:328] Started systemd slice `mesos_executors.slice`
Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.953656 7002
resolver.cpp:69] Creating default secret resolver
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.042822 7002
containerizer.cpp:318] Using isolation { volume/sandbox_path,
volume/host_path, volume/image, network/cni, environment_secret,
cgroups/mem, cgroups/cpu, docker/runtime, filesystem/linux }
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049402 7002
linux_launcher.cpp:144] Using /sys/fs/cgroup/freezer as the freezer
hierarchy for the Linux launcher
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049485 7002
linux_launcher.cpp:176] Using /sys/fs/cgroup/systemd as the systemd
hierarchy for the Linux launcher
Jun 8 08:03:42 mesos mesos-agent[7002]: sh: hadoop: command not found
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.143219 7002
fetcher.cpp:68] Skipping URI fetcher plugin 'hadoop' as it could not be
created: Failed to create HDFS client: Hadoop client is not available,
exit status: 32512
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145257 7002
provisioner.cpp:283] Provisioner backend 'overlay' is not supported on
'/var/lib/mesos/provisioner': Backend 'overlay' is not supported due to
missing d_type support on the underlying filesystem
Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145318 7002
provisioner.cpp:300] Using default backend 'copy'
Jun 8 08:03:42 mesos mesos-agent[7002]: E0608 08:03:42.161227 7002
main.cpp:511] EXIT with status 1: Failed to create a containerizer:
Could not create MesosContainerizer: Failed to clone a sealed file
'/usr/libexec/mesos/mesos-containerizer' in memory: Failed to open
memfd file: Failed to create memfd: Invalid argument
Thanks
Olivier
--
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to start agent 1.9.0: Failed to create memfd error
Posted by Andrei Budnik <ab...@d2iq.com>.
memfd is used to fix CVE-2019-5735 [1] for the Mesos containerizer [2].
This feature is configurable via the `--enable-launcher-sealing`
configuration flag. Mesos packages are built with this feature enabled by
default.
Since memfd requires the latest kernel version, I'd recommend upgrading
your kernel. We've been running CI tests on CentOS 7 with memfd enabled
with no issues so far.
[1] https://lwn.net/Articles/779542/
[2] https://mesos.apache.org/documentation/latest/mesos-containerizer/
On Mon, Jun 8, 2020 at 2:51 PM olivier sallou <ol...@irisa.fr>
wrote:
> On Mon, 2020-06-08 at 13:04 +0200, Andrei Budnik wrote:
> > What is your kernel version?
> >
> > uname -a
> > cat /etc/os-release
>
> Centos 7 based with a fresh yum upgrade
>
> # uname -a
> Linux mesos 3.10.0-327.36.2.el7.x86_64 #1 SMP Mon Oct 10 23:08:37 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@mesos ~]# cat /etc/os-release
> NAME="CentOS Linux"
> VERSION="7 (Core)"
> ID="centos"
> ID_LIKE="rhel fedora"
> VERSION_ID="7"
> PRETTY_NAME="CentOS Linux 7 (Core)"
> ANSI_COLOR="0;31"
> CPE_NAME="cpe:/o:centos:centos:7"
> HOME_URL="https://www.centos.org/"
> BUG_REPORT_URL="https://bugs.centos.org/"
>
> CENTOS_MANTISBT_PROJECT="CentOS-7"
> CENTOS_MANTISBT_PROJECT_VERSION="7"
> REDHAT_SUPPORT_PRODUCT="centos"
> REDHAT_SUPPORT_PRODUCT_VERSION="7"
>
>
> >
> > On Mon, Jun 8, 2020 at 10:13 AM olivier sallou <
> > olivier.sallou@irisa.fr> wrote:
> > > Hi,
> > > I tried to install mesos 1.9.0 (installed from rpm) on CentOS 7. I
> > > configured it for docker with isolation flags but it fails to start
> > > with "Failed to open memfd file: Failed to create memfd: Invalid
> > > argument" error (startup logs below)
> > >
> > >
> > > Jun 8 08:03:41 mesos systemd: Started Mesos Agent.
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.938788
> > > 7002
> > > logging.cpp:201] INFO level logging started!
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939116
> > > 7002
> > > main.cpp:350] Build: 2019-10-22 13:58:29 by
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939124
> > > 7002
> > > main.cpp:351] Version: 1.9.0
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939127
> > > 7002
> > > main.cpp:354] Git tag: 1.9.0-rc3
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939131
> > > 7002
> > > main.cpp:358] Git SHA: 5e79a584e6ec3e9e2f96e8bf418411df9dafac2e
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946902
> > > 7002
> > > systemd.cpp:240] systemd version `219` detected
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946928
> > > 7002
> > > main.cpp:453] Initializing systemd state
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.952776
> > > 7002
> > > systemd.cpp:328] Started systemd slice `mesos_executors.slice`
> > > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.953656
> > > 7002
> > > resolver.cpp:69] Creating default secret resolver
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.042822
> > > 7002
> > > containerizer.cpp:318] Using isolation { volume/sandbox_path,
> > > volume/host_path, volume/image, network/cni, environment_secret,
> > > cgroups/mem, cgroups/cpu, docker/runtime, filesystem/linux }
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049402
> > > 7002
> > > linux_launcher.cpp:144] Using /sys/fs/cgroup/freezer as the freezer
> > > hierarchy for the Linux launcher
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049485
> > > 7002
> > > linux_launcher.cpp:176] Using /sys/fs/cgroup/systemd as the systemd
> > > hierarchy for the Linux launcher
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: sh: hadoop: command not
> > > found
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.143219
> > > 7002
> > > fetcher.cpp:68] Skipping URI fetcher plugin 'hadoop' as it could
> > > not be
> > > created: Failed to create HDFS client: Hadoop client is not
> > > available,
> > > exit status: 32512
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145257
> > > 7002
> > > provisioner.cpp:283] Provisioner backend 'overlay' is not supported
> > > on
> > > '/var/lib/mesos/provisioner': Backend 'overlay' is not supported
> > > due to
> > > missing d_type support on the underlying filesystem
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145318
> > > 7002
> > > provisioner.cpp:300] Using default backend 'copy'
> > > Jun 8 08:03:42 mesos mesos-agent[7002]: E0608 08:03:42.161227
> > > 7002
> > > main.cpp:511] EXIT with status 1: Failed to create a containerizer:
> > > Could not create MesosContainerizer: Failed to clone a sealed file
> > > '/usr/libexec/mesos/mesos-containerizer' in memory: Failed to open
> > > memfd file: Failed to create memfd: Invalid argument
> > >
> > >
> > > Thanks
> > >
> > > Olivier
> > >
> --
> Olivier Sallou
> Univ Rennes, Inria, CNRS, IRISA
> Irisa, Campus de Beaulieu
> F-35042 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: Fail to start agent 1.9.0: Failed to create memfd error
Posted by olivier sallou <ol...@irisa.fr>.
On Mon, 2020-06-08 at 13:04 +0200, Andrei Budnik wrote:
> What is your kernel version?
>
> uname -a
> cat /etc/os-release
Centos 7 based with a fresh yum upgrade
# uname -a
Linux mesos 3.10.0-327.36.2.el7.x86_64 #1 SMP Mon Oct 10 23:08:37 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
[root@mesos ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
>
> On Mon, Jun 8, 2020 at 10:13 AM olivier sallou <
> olivier.sallou@irisa.fr> wrote:
> > Hi,
> > I tried to install mesos 1.9.0 (installed from rpm) on CentOS 7. I
> > configured it for docker with isolation flags but it fails to start
> > with "Failed to open memfd file: Failed to create memfd: Invalid
> > argument" error (startup logs below)
> >
> >
> > Jun 8 08:03:41 mesos systemd: Started Mesos Agent.
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.938788
> > 7002
> > logging.cpp:201] INFO level logging started!
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939116
> > 7002
> > main.cpp:350] Build: 2019-10-22 13:58:29 by
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939124
> > 7002
> > main.cpp:351] Version: 1.9.0
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939127
> > 7002
> > main.cpp:354] Git tag: 1.9.0-rc3
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939131
> > 7002
> > main.cpp:358] Git SHA: 5e79a584e6ec3e9e2f96e8bf418411df9dafac2e
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946902
> > 7002
> > systemd.cpp:240] systemd version `219` detected
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946928
> > 7002
> > main.cpp:453] Initializing systemd state
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.952776
> > 7002
> > systemd.cpp:328] Started systemd slice `mesos_executors.slice`
> > Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.953656
> > 7002
> > resolver.cpp:69] Creating default secret resolver
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.042822
> > 7002
> > containerizer.cpp:318] Using isolation { volume/sandbox_path,
> > volume/host_path, volume/image, network/cni, environment_secret,
> > cgroups/mem, cgroups/cpu, docker/runtime, filesystem/linux }
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049402
> > 7002
> > linux_launcher.cpp:144] Using /sys/fs/cgroup/freezer as the freezer
> > hierarchy for the Linux launcher
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049485
> > 7002
> > linux_launcher.cpp:176] Using /sys/fs/cgroup/systemd as the systemd
> > hierarchy for the Linux launcher
> > Jun 8 08:03:42 mesos mesos-agent[7002]: sh: hadoop: command not
> > found
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.143219
> > 7002
> > fetcher.cpp:68] Skipping URI fetcher plugin 'hadoop' as it could
> > not be
> > created: Failed to create HDFS client: Hadoop client is not
> > available,
> > exit status: 32512
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145257
> > 7002
> > provisioner.cpp:283] Provisioner backend 'overlay' is not supported
> > on
> > '/var/lib/mesos/provisioner': Backend 'overlay' is not supported
> > due to
> > missing d_type support on the underlying filesystem
> > Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145318
> > 7002
> > provisioner.cpp:300] Using default backend 'copy'
> > Jun 8 08:03:42 mesos mesos-agent[7002]: E0608 08:03:42.161227
> > 7002
> > main.cpp:511] EXIT with status 1: Failed to create a containerizer:
> > Could not create MesosContainerizer: Failed to clone a sealed file
> > '/usr/libexec/mesos/mesos-containerizer' in memory: Failed to open
> > memfd file: Failed to create memfd: Invalid argument
> >
> >
> > Thanks
> >
> > Olivier
> >
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Fail to start agent 1.9.0: Failed to create memfd error
Posted by Andrei Budnik <ab...@d2iq.com>.
What is your kernel version?
uname -a
cat /etc/os-release
On Mon, Jun 8, 2020 at 10:13 AM olivier sallou <ol...@irisa.fr>
wrote:
> Hi,
> I tried to install mesos 1.9.0 (installed from rpm) on CentOS 7. I
> configured it for docker with isolation flags but it fails to start
> with "Failed to open memfd file: Failed to create memfd: Invalid
> argument" error (startup logs below)
>
>
> Jun 8 08:03:41 mesos systemd: Started Mesos Agent.
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.938788 7002
> logging.cpp:201] INFO level logging started!
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939116 7002
> main.cpp:350] Build: 2019-10-22 13:58:29 by
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939124 7002
> main.cpp:351] Version: 1.9.0
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939127 7002
> main.cpp:354] Git tag: 1.9.0-rc3
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.939131 7002
> main.cpp:358] Git SHA: 5e79a584e6ec3e9e2f96e8bf418411df9dafac2e
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946902 7002
> systemd.cpp:240] systemd version `219` detected
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.946928 7002
> main.cpp:453] Initializing systemd state
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.952776 7002
> systemd.cpp:328] Started systemd slice `mesos_executors.slice`
> Jun 8 08:03:41 mesos mesos-agent[7002]: I0608 08:03:41.953656 7002
> resolver.cpp:69] Creating default secret resolver
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.042822 7002
> containerizer.cpp:318] Using isolation { volume/sandbox_path,
> volume/host_path, volume/image, network/cni, environment_secret,
> cgroups/mem, cgroups/cpu, docker/runtime, filesystem/linux }
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049402 7002
> linux_launcher.cpp:144] Using /sys/fs/cgroup/freezer as the freezer
> hierarchy for the Linux launcher
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.049485 7002
> linux_launcher.cpp:176] Using /sys/fs/cgroup/systemd as the systemd
> hierarchy for the Linux launcher
> Jun 8 08:03:42 mesos mesos-agent[7002]: sh: hadoop: command not found
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.143219 7002
> fetcher.cpp:68] Skipping URI fetcher plugin 'hadoop' as it could not be
> created: Failed to create HDFS client: Hadoop client is not available,
> exit status: 32512
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145257 7002
> provisioner.cpp:283] Provisioner backend 'overlay' is not supported on
> '/var/lib/mesos/provisioner': Backend 'overlay' is not supported due to
> missing d_type support on the underlying filesystem
> Jun 8 08:03:42 mesos mesos-agent[7002]: I0608 08:03:42.145318 7002
> provisioner.cpp:300] Using default backend 'copy'
> Jun 8 08:03:42 mesos mesos-agent[7002]: E0608 08:03:42.161227 7002
> main.cpp:511] EXIT with status 1: Failed to create a containerizer:
> Could not create MesosContainerizer: Failed to clone a sealed file
> '/usr/libexec/mesos/mesos-containerizer' in memory: Failed to open
> memfd file: Failed to create memfd: Invalid argument
>
>
> Thanks
>
> Olivier
>
> --
>
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>