You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Gary Malouf <ma...@gmail.com> on 2013/07/09 00:41:29 UTC
Mesos fails to start - no logs appear to be written
I've run been running a mesos cluster for about a month in our staging
environment without issue. Today, I tried to compile and install a cluster
in production using the same build process we did for staging but have run
into issues getting it to run from the install location. We have the
log_dir parameter set to /var/log/mesos but no data is showing up in there
or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
To be clear, I am able to run masters and slaves from the directory I built
the code in but the install location seems to have issues.
Two questions:
1) What are the recommended strategies for debugging an install?
2) Any other places where the logs could be showing up?
Thanks for the help.
Gary
Re: Mesos fails to start - no logs appear to be written
Posted by Benjamin Mahler <be...@gmail.com>.
Could you provide the following:
1. How are you installing mesos?
2. How are you starting the master / slave?
3. Can you provide the slave logs?
I'm also unfamiliar with the start-mesos-cluster.sh scripts, I don't see
those in master, what release are running?
Thanks!
Ben
On Mon, Jul 8, 2013 at 3:41 PM, Gary Malouf <ma...@gmail.com> wrote:
> I've run been running a mesos cluster for about a month in our staging
> environment without issue. Today, I tried to compile and install a cluster
> in production using the same build process we did for staging but have run
> into issues getting it to run from the install location. We have the
> log_dir parameter set to /var/log/mesos but no data is showing up in there
> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>
> To be clear, I am able to run masters and slaves from the directory I built
> the code in but the install location seems to have issues.
>
> Two questions:
>
> 1) What are the recommended strategies for debugging an install?
>
> 2) Any other places where the logs could be showing up?
>
>
> Thanks for the help.
>
> Gary
>
Re: Mesos fails to start - no logs appear to be written
Posted by Charles Reiss <wo...@gmail.com>.
On 7/8/13 15:41 , Gary Malouf wrote:
> I've run been running a mesos cluster for about a month in our staging
> environment without issue. Today, I tried to compile and install a cluster
> in production using the same build process we did for staging but have run
> into issues getting it to run from the install location. We have the
> log_dir parameter set to /var/log/mesos but no data is showing up in there
> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>
> To be clear, I am able to run masters and slaves from the directory I built
> the code in but the install location seems to have issues.
>
> Two questions:
>
> 1) What are the recommended strategies for debugging an install?
>
> 2) Any other places where the logs could be showing up?
The mesos-daemon.sh script redirects stdout/err to /dev/null, which is
probably a mistake. Try modifying that to redirect them to a useful file.
- Charles
Fwd: Mesos fails to start - no logs appear to be written
Posted by Gary Malouf <ma...@gmail.com>.
Not sure which address is working at the moment.
---------- Forwarded message ----------
From: Gary Malouf <ma...@gmail.com>
Date: Mon, Jul 8, 2013 at 7:35 PM
Subject: Re: Mesos fails to start - no logs appear to be written
To: dev@mesos.apache.org
1. How are you installing mesos?
- I am using
http://www.globalish.com/am/incubator/mesos/mesos-0.12.0-incubating/mesos-0.12.0-incubating.tar.gzrelease
on CentOS-6.3
- This gets unpacked into a mesos-0.12.0 directory and the I run:
./configure --disable-perftools && make && make install
- I then create /var/log/mesos for output logs
- /usr/local/var/mesos/conf/mesos.conf is created which specifies the zk
addresses of the masters and the log directory
- /usr/local/var/mesos/deploy/masters and
/usr/local/var/mesos/deploy/slaves are created for deploy scripts to know
where to start instances
2. How are you starting the master / slave?
I usually use the mesos-start-cluster.sh script to start the cluster in
staging - I've tried this and others in my new prod environment to no avail.
3. Can you provide the slave logs?
There are no logs being generated from what I can see - they are only
generated when I run from the build location with local scripts as opposed
to the install one.
Hope this helps,
Gary
On Mon, Jul 8, 2013 at 6:41 PM, Gary Malouf <ma...@gmail.com> wrote:
> I've run been running a mesos cluster for about a month in our staging
> environment without issue. Today, I tried to compile and install a cluster
> in production using the same build process we did for staging but have run
> into issues getting it to run from the install location. We have the
> log_dir parameter set to /var/log/mesos but no data is showing up in there
> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>
> To be clear, I am able to run masters and slaves from the directory I
> built the code in but the install location seems to have issues.
>
> Two questions:
>
> 1) What are the recommended strategies for debugging an install?
>
> 2) Any other places where the logs could be showing up?
>
>
> Thanks for the help.
>
> Gary
>
Re: Mesos fails to start - no logs appear to be written
Posted by Gary Malouf <ma...@gmail.com>.
Thanks Ben for the link. The issue was not that the library was not in the
on the path/configured. I checked my /etc/ld.so.conf:
include ld.so.conf.d/*.conf
/usr/local/lib
and the location is definitely there. What needs to happen after install
is that 'ldconfig' needs to be run to update the '/etc/ld.so.cache' to see
the mesos library. Everything works after that command is run. I tried
reading the configure help but was not sure anything would tell 'make
install' to run this after it completes.
-Gary
On Mon, Jul 8, 2013 at 11:45 PM, Gary Malouf <ma...@gmail.com> wrote:
> This is the error message after grabbing standard error:
>
>
> /usr/local/sbin/mesos-master: error while loading shared libraries:
> libmesos-0.12.0.so: cannot open shared object file: No such file or
> directory
>
> What's weird is that the lib is in /usr/local/lib:
>
> bin etc games include java jdk1.7.0_21 lib lib64 libexec sbin
> share src var
> [root@mesos-test-01 local]# ls -la lib
> total 85164
> drwxr-xr-x. 2 root root 4096 Jul 8 16:49 .
> drwxr-xr-x. 14 root root 4096 Jul 8 16:49 ..
> -rwxr-xr-x 1 root root 87194758 Jul 8 16:49 libmesos-0.12.0.so
> -rwxr-xr-x 1 root root 1014 Jul 8 16:49 libmesos.la
> lrwxrwxrwx 1 root root 18 Jul 8 16:49 libmesos.so ->
> libmesos-0.12.0.so
>
>
> On Mon, Jul 8, 2013 at 7:35 PM, Gary Malouf <ma...@gmail.com> wrote:
>
>>
>> 1. How are you installing mesos?
>>
>>
>> - I am using
>> http://www.globalish.com/am/incubator/mesos/mesos-0.12.0-incubating/mesos-0.12.0-incubating.tar.gzrelease on CentOS-6.3
>> - This gets unpacked into a mesos-0.12.0 directory and the I run:
>> ./configure --disable-perftools && make && make install
>> - I then create /var/log/mesos for output logs
>> - /usr/local/var/mesos/conf/mesos.conf is created which specifies the
>> zk addresses of the masters and the log directory
>> - /usr/local/var/mesos/deploy/masters and
>> /usr/local/var/mesos/deploy/slaves are created for deploy scripts to know
>> where to start instances
>>
>>
>>
>> 2. How are you starting the master / slave?
>>
>> I usually use the mesos-start-cluster.sh script to start the cluster in
>> staging - I've tried this and others in my new prod environment to no avail.
>>
>> 3. Can you provide the slave logs?
>>
>> There are no logs being generated from what I can see - they are only
>> generated when I run from the build location with local scripts as opposed
>> to the install one.
>>
>> Hope this helps,
>>
>> Gary
>>
>>
>> On Mon, Jul 8, 2013 at 6:41 PM, Gary Malouf <ma...@gmail.com>wrote:
>>
>>> I've run been running a mesos cluster for about a month in our staging
>>> environment without issue. Today, I tried to compile and install a cluster
>>> in production using the same build process we did for staging but have run
>>> into issues getting it to run from the install location. We have the
>>> log_dir parameter set to /var/log/mesos but no data is showing up in there
>>> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>>>
>>> To be clear, I am able to run masters and slaves from the directory I
>>> built the code in but the install location seems to have issues.
>>>
>>> Two questions:
>>>
>>> 1) What are the recommended strategies for debugging an install?
>>>
>>> 2) Any other places where the logs could be showing up?
>>>
>>>
>>> Thanks for the help.
>>>
>>> Gary
>>>
>>
>>
>
Re: Mesos fails to start - no logs appear to be written
Posted by Benjamin Mahler <be...@gmail.com>.
This may help:
http://stackoverflow.com/questions/4743233/is-usr-local-lib-searched-for-shared-libraries
On Monday, July 8, 2013, Gary Malouf wrote:
> This is the error message after grabbing standard error:
>
>
> /usr/local/sbin/mesos-master: error while loading shared libraries:
> libmesos-0.12.0.so: cannot open shared object file: No such file or
> directory
>
> What's weird is that the lib is in /usr/local/lib:
>
> bin etc games include java jdk1.7.0_21 lib lib64 libexec sbin
> share src var
> [root@mesos-test-01 local]# ls -la lib
> total 85164
> drwxr-xr-x. 2 root root 4096 Jul 8 16:49 .
> drwxr-xr-x. 14 root root 4096 Jul 8 16:49 ..
> -rwxr-xr-x 1 root root 87194758 Jul 8 16:49 libmesos-0.12.0.so
> -rwxr-xr-x 1 root root 1014 Jul 8 16:49 libmesos.la
> lrwxrwxrwx 1 root root 18 Jul 8 16:49 libmesos.so ->
> libmesos-0.12.0.so
>
>
> On Mon, Jul 8, 2013 at 7:35 PM, Gary Malouf <malouf.gary@gmail.com<javascript:;>>
> wrote:
>
> >
> > 1. How are you installing mesos?
> >
> >
> > - I am using
> >
> http://www.globalish.com/am/incubator/mesos/mesos-0.12.0-incubating/mesos-0.12.0-incubating.tar.gzreleaseon CentOS-6.3
> > - This gets unpacked into a mesos-0.12.0 directory and the I run:
> > ./configure --disable-perftools && make && make install
> > - I then create /var/log/mesos for output logs
> > - /usr/local/var/mesos/conf/mesos.conf is created which specifies the
> > zk addresses of the masters and the log directory
> > - /usr/local/var/mesos/deploy/masters and
> > /usr/local/var/mesos/deploy/slaves are created for deploy scripts to
> know
> > where to start instances
> >
> >
> >
> > 2. How are you starting the master / slave?
> >
> > I usually use the mesos-start-cluster.sh script to start the cluster in
> > staging - I've tried this and others in my new prod environment to no
> avail.
> >
> > 3. Can you provide the slave logs?
> >
> > There are no logs being generated from what I can see - they are only
> > generated when I run from the build location with local scripts as
> opposed
> > to the install one.
> >
> > Hope this helps,
> >
> > Gary
> >
> >
> > On Mon, Jul 8, 2013 at 6:41 PM, Gary Malouf <malouf.gary@gmail.com<javascript:;>>
> wrote:
> >
> >> I've run been running a mesos cluster for about a month in our staging
> >> environment without issue. Today, I tried to compile and install a
> cluster
> >> in production using the same build process we did for staging but have
> run
> >> into issues getting it to run from the install location. We have the
> >> log_dir parameter set to /var/log/mesos but no data is showing up in
> there
> >> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
> >>
> >> To be clear, I am able to run masters and slaves from the directory I
> >> built the code in but the install location seems to have issues.
> >>
> >> Two questions:
> >>
> >> 1) What are the recommended strategies for debugging an install?
> >>
> >> 2) Any other places where the logs could be showing up?
> >>
> >>
> >> Thanks for the help.
> >>
> >> Gary
> >>
> >
> >
>
Re: Mesos fails to start - no logs appear to be written
Posted by Gary Malouf <ma...@gmail.com>.
This is the error message after grabbing standard error:
/usr/local/sbin/mesos-master: error while loading shared libraries:
libmesos-0.12.0.so: cannot open shared object file: No such file or
directory
What's weird is that the lib is in /usr/local/lib:
bin etc games include java jdk1.7.0_21 lib lib64 libexec sbin
share src var
[root@mesos-test-01 local]# ls -la lib
total 85164
drwxr-xr-x. 2 root root 4096 Jul 8 16:49 .
drwxr-xr-x. 14 root root 4096 Jul 8 16:49 ..
-rwxr-xr-x 1 root root 87194758 Jul 8 16:49 libmesos-0.12.0.so
-rwxr-xr-x 1 root root 1014 Jul 8 16:49 libmesos.la
lrwxrwxrwx 1 root root 18 Jul 8 16:49 libmesos.so ->
libmesos-0.12.0.so
On Mon, Jul 8, 2013 at 7:35 PM, Gary Malouf <ma...@gmail.com> wrote:
>
> 1. How are you installing mesos?
>
>
> - I am using
> http://www.globalish.com/am/incubator/mesos/mesos-0.12.0-incubating/mesos-0.12.0-incubating.tar.gzrelease on CentOS-6.3
> - This gets unpacked into a mesos-0.12.0 directory and the I run:
> ./configure --disable-perftools && make && make install
> - I then create /var/log/mesos for output logs
> - /usr/local/var/mesos/conf/mesos.conf is created which specifies the
> zk addresses of the masters and the log directory
> - /usr/local/var/mesos/deploy/masters and
> /usr/local/var/mesos/deploy/slaves are created for deploy scripts to know
> where to start instances
>
>
>
> 2. How are you starting the master / slave?
>
> I usually use the mesos-start-cluster.sh script to start the cluster in
> staging - I've tried this and others in my new prod environment to no avail.
>
> 3. Can you provide the slave logs?
>
> There are no logs being generated from what I can see - they are only
> generated when I run from the build location with local scripts as opposed
> to the install one.
>
> Hope this helps,
>
> Gary
>
>
> On Mon, Jul 8, 2013 at 6:41 PM, Gary Malouf <ma...@gmail.com> wrote:
>
>> I've run been running a mesos cluster for about a month in our staging
>> environment without issue. Today, I tried to compile and install a cluster
>> in production using the same build process we did for staging but have run
>> into issues getting it to run from the install location. We have the
>> log_dir parameter set to /var/log/mesos but no data is showing up in there
>> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>>
>> To be clear, I am able to run masters and slaves from the directory I
>> built the code in but the install location seems to have issues.
>>
>> Two questions:
>>
>> 1) What are the recommended strategies for debugging an install?
>>
>> 2) Any other places where the logs could be showing up?
>>
>>
>> Thanks for the help.
>>
>> Gary
>>
>
>
Re: Mesos fails to start - no logs appear to be written
Posted by Gary Malouf <ma...@gmail.com>.
1. How are you installing mesos?
- I am using
http://www.globalish.com/am/incubator/mesos/mesos-0.12.0-incubating/mesos-0.12.0-incubating.tar.gzrelease
on CentOS-6.3
- This gets unpacked into a mesos-0.12.0 directory and the I run:
./configure --disable-perftools && make && make install
- I then create /var/log/mesos for output logs
- /usr/local/var/mesos/conf/mesos.conf is created which specifies the zk
addresses of the masters and the log directory
- /usr/local/var/mesos/deploy/masters and
/usr/local/var/mesos/deploy/slaves are created for deploy scripts to know
where to start instances
2. How are you starting the master / slave?
I usually use the mesos-start-cluster.sh script to start the cluster in
staging - I've tried this and others in my new prod environment to no avail.
3. Can you provide the slave logs?
There are no logs being generated from what I can see - they are only
generated when I run from the build location with local scripts as opposed
to the install one.
Hope this helps,
Gary
On Mon, Jul 8, 2013 at 6:41 PM, Gary Malouf <ma...@gmail.com> wrote:
> I've run been running a mesos cluster for about a month in our staging
> environment without issue. Today, I tried to compile and install a cluster
> in production using the same build process we did for staging but have run
> into issues getting it to run from the install location. We have the
> log_dir parameter set to /var/log/mesos but no data is showing up in there
> or in /tmp when I run the sbin/start-mesos-cluster.sh scripts.
>
> To be clear, I am able to run masters and slaves from the directory I
> built the code in but the install location seems to have issues.
>
> Two questions:
>
> 1) What are the recommended strategies for debugging an install?
>
> 2) Any other places where the logs could be showing up?
>
>
> Thanks for the help.
>
> Gary
>