You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by praveenesh kumar <pr...@gmail.com> on 2012/09/19 08:57:02 UTC
Problems in starting mesos
Hi all,
I have been able to successfully build and install mesos on my physical
nodes. I can see the mesos web-url with list of mesos slaves. However I am
finding few difficulties w.r.t to following things:
*1. How to start mesos-master.sh and mesos-slave.sh as a background process
? *
I am seeing all my output on the console. Since I want to remotely start
the service on the cluster, I tried using nohup. Its working for now, but
is it the right way? I am hoping there should be better way to do that.
*2.* *Whatever memory I am seeing on the mesos-web url, is it the total
memory or the free memory available on my system ?*
I am seeing very low memory (0.001 GB) -- Does it indicate the current
available memory ?
3. I have enable the "log_dir" directory, while starting the mesos-master
and slaves. I can see the logs for my slave machines from mesos web url,
but not able to see logs for my master. Its saying *"Failed to Initialize
.... Retrying"*. I can see work directory is there as well as the logs are
also there. Set the permission to 777 also. Still no success.
Any help on the above to understand mesos, would be very helpful.
Regards,
Praveenesh
Re: Problems in starting mesos
Posted by Benjamin Mahler <bm...@twitter.com>.
So the master/slave attach the log file when initializing. Therefore,
unless you're faster than the master, it will try to attach the log file
before you can create your symlink and will fail.
Can you please provide me the contents of the mesos-master.sh /
mesos-slave.sh that you're running? Where did you get these, directly from
trunk?
Other than that, just watch for https://reviews.apache.org/r/7061/ getting
submitted.
On Tue, Sep 25, 2012 at 3:14 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi Benjamin,
>
> I changed the file permissions to 777 for my work directories and log
> files. Still I don't see any server logs in webui.
> Also I am using mesos-slaves.sh and mesos-master.sh command to start the
> master and slave ( so I am not running binaries directly).
>
> Only way I can see my slave logs is my creating the links ( the thing that
> I was doing before).
> But I can't see master logs by anyway. I am getting the same error with and
> without links.
>
> Regards,
> Praveenesh
>
>
>
> On Tue, Sep 25, 2012 at 12:05 AM, Benjamin Mahler <bmahler@twitter.com
> >wrote:
>
> > Ah, I was confused by the asterisks in your commands.
> >
> > So you're running:
> > $ ln -s lt-mesos-slave.INFO mesos-slave.INFO
> > because your log file is named lt-mesos-slave.INFO?
> >
> > This suggests you're running the lt-mesos-slave binary directly, instead
> of
> > mesos-slave.sh. Google-glog will use basename(argv0) as the logging
> > destination, so running mesos-slave.sh will produce a log file named
> > mesos-slave.INFO.
> >
> > Regardless, I have a change that should fix this issue for you:
> > https://reviews.apache.org/r/7061/
> > It's not currently in trunk, so in the interim I would suggest looking at
> > permissions of the work directory you created on the master, to ensure
> it's
> > readable by the master process.
> >
> > Hope this helps!
> >
> > On Mon, Sep 24, 2012 at 3:32 AM, praveenesh kumar <praveenesh@gmail.com
> > >wrote:
> >
> > > Hi Benjamin,
> > >
> > > So I did, what you suggest:
> > >
> > > 1. -wipe your existing symbolic links, etc
> > > -remove the ln commands from your workflow
> > > -run again, and see if the webui master LOG works, if not please
> > attach
> > > the master logs and the commands you ran
> > >
> > > 2. I can't still see the webui master Logs...On web UI, I am seeing
> > "Failed
> > > to initialize .. retrying"
> > >
> > > Error :
> > > E0924 05:10:30.077483 10025 files.cpp:92] Error attaching path
> > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into
> an
> > > absolute path
> > >
> > >
> > > Its the same error as I was getting before.
> > >
> > > 3. I deleted the links I created for slave nodes also. When I start
> slave
> > > nodes, now I am not able to see the slave node logs also.
> > >
> > > * tail: cannot open
> > > `/usr/local/mesos-spark/mesos/work/mesos-slave.INFO' for reading: No
> such
> > > file or directory*
> > >
> > > Slave error logs --
> > >
> > > E0924 05:21:00.418911 11332 files.cpp:92] Error attaching path
> > > '/usr/local/mesos-spark/mesos/work/mesos-slave.INFO': Failed to
> > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO into an
> > > absolute path
> > >
> > > E0924 05:21:00.525163 11332 slave.cpp:362] Failed to attach log file:
> > > Failed to canonicalize
> /usr/local/mesos-spark/mesos/work/mesos-slave.INFO
> > > into an absolute path
> > >
> > >
> > > After reading the above errors only, I created those symlinks and it
> > > allowed me to see slave logs, so I thought of doing the same for master
> > > also, but couldn't succeed.
> > >
> > > Thanks for any help.
> > >
> > > Regards,
> > > Praveenesh
> > >
> > >
> > > On Fri, Sep 21, 2012 at 10:40 PM, Benjamin Mahler <bmahler@twitter.com
> > > >wrote:
> > >
> > > > 1. I would suggest looking into the 'daemon' linux command man page.
> > > >
> > > > 3. What are you trying to accomplish with the symbolic links? They
> > appear
> > > > to conflict with the logging files google-glog will create. The error
> > > from
> > > > the master log indicates that the log files cannot be attached (and
> > hence
> > > > viewing logs from the webui won't work).
> > > >
> > > > I would suggest:
> > > > -wipe your existing symbolic links, etc
> > > > -remove the ln commands from your workflow
> > > > -run again, and see if the webui master LOG works, if not please
> > attach
> > > > the master logs and the commands you ran
> > > >
> > > > Hope this helps!
> > > >
> > > > On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar <
> > praveenesh@gmail.com
> > > > >wrote:
> > > >
> > > > > Thanks Benjamin.
> > > > >
> > > > > 1. I was wondering how can I run mesos-master and slaves as daemon
> > > > process.
> > > > > Nohup is giving me issues to launch them from remotely.
> > > > >
> > > > > 3. I have build mesos from trunk and starting meso master and slave
> > by
> > > > ---
> > > > >
> > > > > Mesos-master
> > > > >
> > > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
> > > > > --log_dir=/usr/local/mesos-spark/mesos/work >>
> /tmp/mesos-master.txt
> > &*
> > > > > *
> > > > > *
> > > > > *ln -s lt-mesos-master.INFO mesos-master.INFO*
> > > > > *ln -s lt-mesos-master.ERROR mesos-master.ERROR*
> > > > > *ln -s lt-mesos-master.WARNING mesos-master.WARNING*
> > > > >
> > > > >
> > > > >
> > > > > Mesos-slave --- (Note I need to create links to all log files to
> > > enable
> > > > > logging from mesos web url)
> > > > >
> > > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh
> > > --master=br9:5050
> > > > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt
> > &*
> > > > >
> > > > > *cd /usr/local/mesos-spark/mesos/*
> > > > > *chmod 777 -R work*
> > > > > *cd work*
> > > > > *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
> > > > > *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
> > > > > *ln -s lt-mesos-slave.INFO mesos-slave.INFO*
> > > > >
> > > > >
> > > > > I see only these suspicious lines in lt-mesos-master.ERROR
> > > > >
> > > > > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > > > > E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
> > > > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > > > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> into
> > > an
> > > > > absolute path
> > > > > E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log
> > file:
> > > > > Failed to canonicalize
> > > > /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> > > > > into an absolute path
> > > > >
> > > > > Regards,
> > > > > Praveenesh
> > > > >
> > > > >
> > > > > On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <
> > bmahler@twitter.com
> > > > > >wrote:
> > > > >
> > > > > > 1. You could run it as a daemon, what you're doing with nohup is
> > fine
> > > > as
> > > > > > well.
> > > > > >
> > > > > > 2. The webui shows various resource statistics: [image: Inline
> > image
> > > 2]
> > > > > >
> > > > > > *Total memory* indicates the total amount of memory available on
> > all
> > > > the
> > > > > > slaves.
> > > > > > Total Memory should equal Used + Idle + Offered
> > > > > > *
> > > > > > *
> > > > > > *Used:* Memory used by the frameworks.
> > > > > > *Offered:* This memory has been offered but the framework hasn't
> > > > > > responded yet.
> > > > > > *Idle:* Any remaining memory in the system.
> > > > > >
> > > > > > 3. Are you running off trunk or a release? What is the command
> line
> > > > > you're
> > > > > > running to start the master? Can you provide the master's log
> > output?
> > > > > >
> > > > > > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <
> > > > praveenesh@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I have been able to successfully build and install mesos on my
> > > > physical
> > > > > >> nodes. I can see the mesos web-url with list of mesos slaves.
> > > However
> > > > I
> > > > > am
> > > > > >> finding few difficulties w.r.t to following things:
> > > > > >>
> > > > > >> *1. How to start mesos-master.sh and mesos-slave.sh as a
> > background
> > > > > >> process
> > > > > >>
> > > > > >> ? *
> > > > > >> I am seeing all my output on the console. Since I want to
> remotely
> > > > start
> > > > > >> the service on the cluster, I tried using nohup. Its working for
> > > now,
> > > > > but
> > > > > >> is it the right way? I am hoping there should be better way to
> do
> > > > that.
> > > > > >>
> > > > > >> *2.* *Whatever memory I am seeing on the mesos-web url, is it
> the
> > > > total
> > > > > >> memory or the free memory available on my system ?*
> > > > > >>
> > > > > >> I am seeing very low memory (0.001 GB) -- Does it indicate the
> > > current
> > > > > >> available memory ?
> > > > > >>
> > > > > >> 3. I have enable the "log_dir" directory, while starting the
> > > > > mesos-master
> > > > > >> and slaves. I can see the logs for my slave machines from mesos
> > web
> > > > url,
> > > > > >> but not able to see logs for my master. Its saying *"Failed to
> > > > > Initialize
> > > > > >> .... Retrying"*. I can see work directory is there as well as
> the
> > > logs
> > > > > are
> > > > > >>
> > > > > >> also there. Set the permission to 777 also. Still no success.
> > > > > >>
> > > > > >>
> > > > > >> Any help on the above to understand mesos, would be very
> helpful.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Praveenesh
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Problems in starting mesos
Posted by praveenesh kumar <pr...@gmail.com>.
Hi Benjamin,
I changed the file permissions to 777 for my work directories and log
files. Still I don't see any server logs in webui.
Also I am using mesos-slaves.sh and mesos-master.sh command to start the
master and slave ( so I am not running binaries directly).
Only way I can see my slave logs is my creating the links ( the thing that
I was doing before).
But I can't see master logs by anyway. I am getting the same error with and
without links.
Regards,
Praveenesh
On Tue, Sep 25, 2012 at 12:05 AM, Benjamin Mahler <bm...@twitter.com>wrote:
> Ah, I was confused by the asterisks in your commands.
>
> So you're running:
> $ ln -s lt-mesos-slave.INFO mesos-slave.INFO
> because your log file is named lt-mesos-slave.INFO?
>
> This suggests you're running the lt-mesos-slave binary directly, instead of
> mesos-slave.sh. Google-glog will use basename(argv0) as the logging
> destination, so running mesos-slave.sh will produce a log file named
> mesos-slave.INFO.
>
> Regardless, I have a change that should fix this issue for you:
> https://reviews.apache.org/r/7061/
> It's not currently in trunk, so in the interim I would suggest looking at
> permissions of the work directory you created on the master, to ensure it's
> readable by the master process.
>
> Hope this helps!
>
> On Mon, Sep 24, 2012 at 3:32 AM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
>
> > Hi Benjamin,
> >
> > So I did, what you suggest:
> >
> > 1. -wipe your existing symbolic links, etc
> > -remove the ln commands from your workflow
> > -run again, and see if the webui master LOG works, if not please
> attach
> > the master logs and the commands you ran
> >
> > 2. I can't still see the webui master Logs...On web UI, I am seeing
> "Failed
> > to initialize .. retrying"
> >
> > Error :
> > E0924 05:10:30.077483 10025 files.cpp:92] Error attaching path
> > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
> > absolute path
> >
> >
> > Its the same error as I was getting before.
> >
> > 3. I deleted the links I created for slave nodes also. When I start slave
> > nodes, now I am not able to see the slave node logs also.
> >
> > * tail: cannot open
> > `/usr/local/mesos-spark/mesos/work/mesos-slave.INFO' for reading: No such
> > file or directory*
> >
> > Slave error logs --
> >
> > E0924 05:21:00.418911 11332 files.cpp:92] Error attaching path
> > '/usr/local/mesos-spark/mesos/work/mesos-slave.INFO': Failed to
> > canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO into an
> > absolute path
> >
> > E0924 05:21:00.525163 11332 slave.cpp:362] Failed to attach log file:
> > Failed to canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO
> > into an absolute path
> >
> >
> > After reading the above errors only, I created those symlinks and it
> > allowed me to see slave logs, so I thought of doing the same for master
> > also, but couldn't succeed.
> >
> > Thanks for any help.
> >
> > Regards,
> > Praveenesh
> >
> >
> > On Fri, Sep 21, 2012 at 10:40 PM, Benjamin Mahler <bmahler@twitter.com
> > >wrote:
> >
> > > 1. I would suggest looking into the 'daemon' linux command man page.
> > >
> > > 3. What are you trying to accomplish with the symbolic links? They
> appear
> > > to conflict with the logging files google-glog will create. The error
> > from
> > > the master log indicates that the log files cannot be attached (and
> hence
> > > viewing logs from the webui won't work).
> > >
> > > I would suggest:
> > > -wipe your existing symbolic links, etc
> > > -remove the ln commands from your workflow
> > > -run again, and see if the webui master LOG works, if not please
> attach
> > > the master logs and the commands you ran
> > >
> > > Hope this helps!
> > >
> > > On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar <
> praveenesh@gmail.com
> > > >wrote:
> > >
> > > > Thanks Benjamin.
> > > >
> > > > 1. I was wondering how can I run mesos-master and slaves as daemon
> > > process.
> > > > Nohup is giving me issues to launch them from remotely.
> > > >
> > > > 3. I have build mesos from trunk and starting meso master and slave
> by
> > > ---
> > > >
> > > > Mesos-master
> > > >
> > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
> > > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-master.txt
> &*
> > > > *
> > > > *
> > > > *ln -s lt-mesos-master.INFO mesos-master.INFO*
> > > > *ln -s lt-mesos-master.ERROR mesos-master.ERROR*
> > > > *ln -s lt-mesos-master.WARNING mesos-master.WARNING*
> > > >
> > > >
> > > >
> > > > Mesos-slave --- (Note I need to create links to all log files to
> > enable
> > > > logging from mesos web url)
> > > >
> > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh
> > --master=br9:5050
> > > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt
> &*
> > > >
> > > > *cd /usr/local/mesos-spark/mesos/*
> > > > *chmod 777 -R work*
> > > > *cd work*
> > > > *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
> > > > *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
> > > > *ln -s lt-mesos-slave.INFO mesos-slave.INFO*
> > > >
> > > >
> > > > I see only these suspicious lines in lt-mesos-master.ERROR
> > > >
> > > > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > > > E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
> > > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into
> > an
> > > > absolute path
> > > > E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log
> file:
> > > > Failed to canonicalize
> > > /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> > > > into an absolute path
> > > >
> > > > Regards,
> > > > Praveenesh
> > > >
> > > >
> > > > On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <
> bmahler@twitter.com
> > > > >wrote:
> > > >
> > > > > 1. You could run it as a daemon, what you're doing with nohup is
> fine
> > > as
> > > > > well.
> > > > >
> > > > > 2. The webui shows various resource statistics: [image: Inline
> image
> > 2]
> > > > >
> > > > > *Total memory* indicates the total amount of memory available on
> all
> > > the
> > > > > slaves.
> > > > > Total Memory should equal Used + Idle + Offered
> > > > > *
> > > > > *
> > > > > *Used:* Memory used by the frameworks.
> > > > > *Offered:* This memory has been offered but the framework hasn't
> > > > > responded yet.
> > > > > *Idle:* Any remaining memory in the system.
> > > > >
> > > > > 3. Are you running off trunk or a release? What is the command line
> > > > you're
> > > > > running to start the master? Can you provide the master's log
> output?
> > > > >
> > > > > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <
> > > praveenesh@gmail.com
> > > > >wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I have been able to successfully build and install mesos on my
> > > physical
> > > > >> nodes. I can see the mesos web-url with list of mesos slaves.
> > However
> > > I
> > > > am
> > > > >> finding few difficulties w.r.t to following things:
> > > > >>
> > > > >> *1. How to start mesos-master.sh and mesos-slave.sh as a
> background
> > > > >> process
> > > > >>
> > > > >> ? *
> > > > >> I am seeing all my output on the console. Since I want to remotely
> > > start
> > > > >> the service on the cluster, I tried using nohup. Its working for
> > now,
> > > > but
> > > > >> is it the right way? I am hoping there should be better way to do
> > > that.
> > > > >>
> > > > >> *2.* *Whatever memory I am seeing on the mesos-web url, is it the
> > > total
> > > > >> memory or the free memory available on my system ?*
> > > > >>
> > > > >> I am seeing very low memory (0.001 GB) -- Does it indicate the
> > current
> > > > >> available memory ?
> > > > >>
> > > > >> 3. I have enable the "log_dir" directory, while starting the
> > > > mesos-master
> > > > >> and slaves. I can see the logs for my slave machines from mesos
> web
> > > url,
> > > > >> but not able to see logs for my master. Its saying *"Failed to
> > > > Initialize
> > > > >> .... Retrying"*. I can see work directory is there as well as the
> > logs
> > > > are
> > > > >>
> > > > >> also there. Set the permission to 777 also. Still no success.
> > > > >>
> > > > >>
> > > > >> Any help on the above to understand mesos, would be very helpful.
> > > > >>
> > > > >> Regards,
> > > > >> Praveenesh
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Problems in starting mesos
Posted by Benjamin Mahler <bm...@twitter.com>.
Ah, I was confused by the asterisks in your commands.
So you're running:
$ ln -s lt-mesos-slave.INFO mesos-slave.INFO
because your log file is named lt-mesos-slave.INFO?
This suggests you're running the lt-mesos-slave binary directly, instead of
mesos-slave.sh. Google-glog will use basename(argv0) as the logging
destination, so running mesos-slave.sh will produce a log file named
mesos-slave.INFO.
Regardless, I have a change that should fix this issue for you:
https://reviews.apache.org/r/7061/
It's not currently in trunk, so in the interim I would suggest looking at
permissions of the work directory you created on the master, to ensure it's
readable by the master process.
Hope this helps!
On Mon, Sep 24, 2012 at 3:32 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi Benjamin,
>
> So I did, what you suggest:
>
> 1. -wipe your existing symbolic links, etc
> -remove the ln commands from your workflow
> -run again, and see if the webui master LOG works, if not please attach
> the master logs and the commands you ran
>
> 2. I can't still see the webui master Logs...On web UI, I am seeing "Failed
> to initialize .. retrying"
>
> Error :
> E0924 05:10:30.077483 10025 files.cpp:92] Error attaching path
> '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
> absolute path
>
>
> Its the same error as I was getting before.
>
> 3. I deleted the links I created for slave nodes also. When I start slave
> nodes, now I am not able to see the slave node logs also.
>
> * tail: cannot open
> `/usr/local/mesos-spark/mesos/work/mesos-slave.INFO' for reading: No such
> file or directory*
>
> Slave error logs --
>
> E0924 05:21:00.418911 11332 files.cpp:92] Error attaching path
> '/usr/local/mesos-spark/mesos/work/mesos-slave.INFO': Failed to
> canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO into an
> absolute path
>
> E0924 05:21:00.525163 11332 slave.cpp:362] Failed to attach log file:
> Failed to canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO
> into an absolute path
>
>
> After reading the above errors only, I created those symlinks and it
> allowed me to see slave logs, so I thought of doing the same for master
> also, but couldn't succeed.
>
> Thanks for any help.
>
> Regards,
> Praveenesh
>
>
> On Fri, Sep 21, 2012 at 10:40 PM, Benjamin Mahler <bmahler@twitter.com
> >wrote:
>
> > 1. I would suggest looking into the 'daemon' linux command man page.
> >
> > 3. What are you trying to accomplish with the symbolic links? They appear
> > to conflict with the logging files google-glog will create. The error
> from
> > the master log indicates that the log files cannot be attached (and hence
> > viewing logs from the webui won't work).
> >
> > I would suggest:
> > -wipe your existing symbolic links, etc
> > -remove the ln commands from your workflow
> > -run again, and see if the webui master LOG works, if not please attach
> > the master logs and the commands you ran
> >
> > Hope this helps!
> >
> > On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar <praveenesh@gmail.com
> > >wrote:
> >
> > > Thanks Benjamin.
> > >
> > > 1. I was wondering how can I run mesos-master and slaves as daemon
> > process.
> > > Nohup is giving me issues to launch them from remotely.
> > >
> > > 3. I have build mesos from trunk and starting meso master and slave by
> > ---
> > >
> > > Mesos-master
> > >
> > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
> > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-master.txt &*
> > > *
> > > *
> > > *ln -s lt-mesos-master.INFO mesos-master.INFO*
> > > *ln -s lt-mesos-master.ERROR mesos-master.ERROR*
> > > *ln -s lt-mesos-master.WARNING mesos-master.WARNING*
> > >
> > >
> > >
> > > Mesos-slave --- (Note I need to create links to all log files to
> enable
> > > logging from mesos web url)
> > >
> > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh
> --master=br9:5050
> > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt &*
> > >
> > > *cd /usr/local/mesos-spark/mesos/*
> > > *chmod 777 -R work*
> > > *cd work*
> > > *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
> > > *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
> > > *ln -s lt-mesos-slave.INFO mesos-slave.INFO*
> > >
> > >
> > > I see only these suspicious lines in lt-mesos-master.ERROR
> > >
> > > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > > E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
> > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into
> an
> > > absolute path
> > > E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log file:
> > > Failed to canonicalize
> > /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> > > into an absolute path
> > >
> > > Regards,
> > > Praveenesh
> > >
> > >
> > > On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <bmahler@twitter.com
> > > >wrote:
> > >
> > > > 1. You could run it as a daemon, what you're doing with nohup is fine
> > as
> > > > well.
> > > >
> > > > 2. The webui shows various resource statistics: [image: Inline image
> 2]
> > > >
> > > > *Total memory* indicates the total amount of memory available on all
> > the
> > > > slaves.
> > > > Total Memory should equal Used + Idle + Offered
> > > > *
> > > > *
> > > > *Used:* Memory used by the frameworks.
> > > > *Offered:* This memory has been offered but the framework hasn't
> > > > responded yet.
> > > > *Idle:* Any remaining memory in the system.
> > > >
> > > > 3. Are you running off trunk or a release? What is the command line
> > > you're
> > > > running to start the master? Can you provide the master's log output?
> > > >
> > > > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <
> > praveenesh@gmail.com
> > > >wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I have been able to successfully build and install mesos on my
> > physical
> > > >> nodes. I can see the mesos web-url with list of mesos slaves.
> However
> > I
> > > am
> > > >> finding few difficulties w.r.t to following things:
> > > >>
> > > >> *1. How to start mesos-master.sh and mesos-slave.sh as a background
> > > >> process
> > > >>
> > > >> ? *
> > > >> I am seeing all my output on the console. Since I want to remotely
> > start
> > > >> the service on the cluster, I tried using nohup. Its working for
> now,
> > > but
> > > >> is it the right way? I am hoping there should be better way to do
> > that.
> > > >>
> > > >> *2.* *Whatever memory I am seeing on the mesos-web url, is it the
> > total
> > > >> memory or the free memory available on my system ?*
> > > >>
> > > >> I am seeing very low memory (0.001 GB) -- Does it indicate the
> current
> > > >> available memory ?
> > > >>
> > > >> 3. I have enable the "log_dir" directory, while starting the
> > > mesos-master
> > > >> and slaves. I can see the logs for my slave machines from mesos web
> > url,
> > > >> but not able to see logs for my master. Its saying *"Failed to
> > > Initialize
> > > >> .... Retrying"*. I can see work directory is there as well as the
> logs
> > > are
> > > >>
> > > >> also there. Set the permission to 777 also. Still no success.
> > > >>
> > > >>
> > > >> Any help on the above to understand mesos, would be very helpful.
> > > >>
> > > >> Regards,
> > > >> Praveenesh
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Problems in starting mesos
Posted by praveenesh kumar <pr...@gmail.com>.
Hi Benjamin,
So I did, what you suggest:
1. -wipe your existing symbolic links, etc
-remove the ln commands from your workflow
-run again, and see if the webui master LOG works, if not please attach
the master logs and the commands you ran
2. I can't still see the webui master Logs...On web UI, I am seeing "Failed
to initialize .. retrying"
Error :
E0924 05:10:30.077483 10025 files.cpp:92] Error attaching path
'/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
absolute path
Its the same error as I was getting before.
3. I deleted the links I created for slave nodes also. When I start slave
nodes, now I am not able to see the slave node logs also.
* tail: cannot open
`/usr/local/mesos-spark/mesos/work/mesos-slave.INFO' for reading: No such
file or directory*
Slave error logs --
E0924 05:21:00.418911 11332 files.cpp:92] Error attaching path
'/usr/local/mesos-spark/mesos/work/mesos-slave.INFO': Failed to
canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO into an
absolute path
E0924 05:21:00.525163 11332 slave.cpp:362] Failed to attach log file:
Failed to canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO
into an absolute path
After reading the above errors only, I created those symlinks and it
allowed me to see slave logs, so I thought of doing the same for master
also, but couldn't succeed.
Thanks for any help.
Regards,
Praveenesh
On Fri, Sep 21, 2012 at 10:40 PM, Benjamin Mahler <bm...@twitter.com>wrote:
> 1. I would suggest looking into the 'daemon' linux command man page.
>
> 3. What are you trying to accomplish with the symbolic links? They appear
> to conflict with the logging files google-glog will create. The error from
> the master log indicates that the log files cannot be attached (and hence
> viewing logs from the webui won't work).
>
> I would suggest:
> -wipe your existing symbolic links, etc
> -remove the ln commands from your workflow
> -run again, and see if the webui master LOG works, if not please attach
> the master logs and the commands you ran
>
> Hope this helps!
>
> On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
>
> > Thanks Benjamin.
> >
> > 1. I was wondering how can I run mesos-master and slaves as daemon
> process.
> > Nohup is giving me issues to launch them from remotely.
> >
> > 3. I have build mesos from trunk and starting meso master and slave by
> ---
> >
> > Mesos-master
> >
> > *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
> > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-master.txt &*
> > *
> > *
> > *ln -s lt-mesos-master.INFO mesos-master.INFO*
> > *ln -s lt-mesos-master.ERROR mesos-master.ERROR*
> > *ln -s lt-mesos-master.WARNING mesos-master.WARNING*
> >
> >
> >
> > Mesos-slave --- (Note I need to create links to all log files to enable
> > logging from mesos web url)
> >
> > *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh --master=br9:5050
> > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt &*
> >
> > *cd /usr/local/mesos-spark/mesos/*
> > *chmod 777 -R work*
> > *cd work*
> > *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
> > *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
> > *ln -s lt-mesos-slave.INFO mesos-slave.INFO*
> >
> >
> > I see only these suspicious lines in lt-mesos-master.ERROR
> >
> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
> > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
> > absolute path
> > E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log file:
> > Failed to canonicalize
> /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> > into an absolute path
> >
> > Regards,
> > Praveenesh
> >
> >
> > On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <bmahler@twitter.com
> > >wrote:
> >
> > > 1. You could run it as a daemon, what you're doing with nohup is fine
> as
> > > well.
> > >
> > > 2. The webui shows various resource statistics: [image: Inline image 2]
> > >
> > > *Total memory* indicates the total amount of memory available on all
> the
> > > slaves.
> > > Total Memory should equal Used + Idle + Offered
> > > *
> > > *
> > > *Used:* Memory used by the frameworks.
> > > *Offered:* This memory has been offered but the framework hasn't
> > > responded yet.
> > > *Idle:* Any remaining memory in the system.
> > >
> > > 3. Are you running off trunk or a release? What is the command line
> > you're
> > > running to start the master? Can you provide the master's log output?
> > >
> > > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <
> praveenesh@gmail.com
> > >wrote:
> > >
> > >> Hi all,
> > >>
> > >> I have been able to successfully build and install mesos on my
> physical
> > >> nodes. I can see the mesos web-url with list of mesos slaves. However
> I
> > am
> > >> finding few difficulties w.r.t to following things:
> > >>
> > >> *1. How to start mesos-master.sh and mesos-slave.sh as a background
> > >> process
> > >>
> > >> ? *
> > >> I am seeing all my output on the console. Since I want to remotely
> start
> > >> the service on the cluster, I tried using nohup. Its working for now,
> > but
> > >> is it the right way? I am hoping there should be better way to do
> that.
> > >>
> > >> *2.* *Whatever memory I am seeing on the mesos-web url, is it the
> total
> > >> memory or the free memory available on my system ?*
> > >>
> > >> I am seeing very low memory (0.001 GB) -- Does it indicate the current
> > >> available memory ?
> > >>
> > >> 3. I have enable the "log_dir" directory, while starting the
> > mesos-master
> > >> and slaves. I can see the logs for my slave machines from mesos web
> url,
> > >> but not able to see logs for my master. Its saying *"Failed to
> > Initialize
> > >> .... Retrying"*. I can see work directory is there as well as the logs
> > are
> > >>
> > >> also there. Set the permission to 777 also. Still no success.
> > >>
> > >>
> > >> Any help on the above to understand mesos, would be very helpful.
> > >>
> > >> Regards,
> > >> Praveenesh
> > >>
> > >
> > >
> >
>
Re: Problems in starting mesos
Posted by Benjamin Mahler <bm...@twitter.com>.
1. I would suggest looking into the 'daemon' linux command man page.
3. What are you trying to accomplish with the symbolic links? They appear
to conflict with the logging files google-glog will create. The error from
the master log indicates that the log files cannot be attached (and hence
viewing logs from the webui won't work).
I would suggest:
-wipe your existing symbolic links, etc
-remove the ln commands from your workflow
-run again, and see if the webui master LOG works, if not please attach
the master logs and the commands you ran
Hope this helps!
On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar <pr...@gmail.com>wrote:
> Thanks Benjamin.
>
> 1. I was wondering how can I run mesos-master and slaves as daemon process.
> Nohup is giving me issues to launch them from remotely.
>
> 3. I have build mesos from trunk and starting meso master and slave by ---
>
> Mesos-master
>
> *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
> --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-master.txt &*
> *
> *
> *ln -s lt-mesos-master.INFO mesos-master.INFO*
> *ln -s lt-mesos-master.ERROR mesos-master.ERROR*
> *ln -s lt-mesos-master.WARNING mesos-master.WARNING*
>
>
>
> Mesos-slave --- (Note I need to create links to all log files to enable
> logging from mesos web url)
>
> *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh --master=br9:5050
> --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt &*
>
> *cd /usr/local/mesos-spark/mesos/*
> *chmod 777 -R work*
> *cd work*
> *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
> *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
> *ln -s lt-mesos-slave.INFO mesos-slave.INFO*
>
>
> I see only these suspicious lines in lt-mesos-master.ERROR
>
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
> '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
> canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
> absolute path
> E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log file:
> Failed to canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO
> into an absolute path
>
> Regards,
> Praveenesh
>
>
> On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <bmahler@twitter.com
> >wrote:
>
> > 1. You could run it as a daemon, what you're doing with nohup is fine as
> > well.
> >
> > 2. The webui shows various resource statistics: [image: Inline image 2]
> >
> > *Total memory* indicates the total amount of memory available on all the
> > slaves.
> > Total Memory should equal Used + Idle + Offered
> > *
> > *
> > *Used:* Memory used by the frameworks.
> > *Offered:* This memory has been offered but the framework hasn't
> > responded yet.
> > *Idle:* Any remaining memory in the system.
> >
> > 3. Are you running off trunk or a release? What is the command line
> you're
> > running to start the master? Can you provide the master's log output?
> >
> > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
> >
> >> Hi all,
> >>
> >> I have been able to successfully build and install mesos on my physical
> >> nodes. I can see the mesos web-url with list of mesos slaves. However I
> am
> >> finding few difficulties w.r.t to following things:
> >>
> >> *1. How to start mesos-master.sh and mesos-slave.sh as a background
> >> process
> >>
> >> ? *
> >> I am seeing all my output on the console. Since I want to remotely start
> >> the service on the cluster, I tried using nohup. Its working for now,
> but
> >> is it the right way? I am hoping there should be better way to do that.
> >>
> >> *2.* *Whatever memory I am seeing on the mesos-web url, is it the total
> >> memory or the free memory available on my system ?*
> >>
> >> I am seeing very low memory (0.001 GB) -- Does it indicate the current
> >> available memory ?
> >>
> >> 3. I have enable the "log_dir" directory, while starting the
> mesos-master
> >> and slaves. I can see the logs for my slave machines from mesos web url,
> >> but not able to see logs for my master. Its saying *"Failed to
> Initialize
> >> .... Retrying"*. I can see work directory is there as well as the logs
> are
> >>
> >> also there. Set the permission to 777 also. Still no success.
> >>
> >>
> >> Any help on the above to understand mesos, would be very helpful.
> >>
> >> Regards,
> >> Praveenesh
> >>
> >
> >
>
Re: Problems in starting mesos
Posted by praveenesh kumar <pr...@gmail.com>.
Thanks Benjamin.
1. I was wondering how can I run mesos-master and slaves as daemon process.
Nohup is giving me issues to launch them from remotely.
3. I have build mesos from trunk and starting meso master and slave by ---
Mesos-master
*nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh
--log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-master.txt &*
*
*
*ln -s lt-mesos-master.INFO mesos-master.INFO*
*ln -s lt-mesos-master.ERROR mesos-master.ERROR*
*ln -s lt-mesos-master.WARNING mesos-master.WARNING*
Mesos-slave --- (Note I need to create links to all log files to enable
logging from mesos web url)
*nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh --master=br9:5050
--log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt &*
*cd /usr/local/mesos-spark/mesos/*
*chmod 777 -R work*
*cd work*
*ln -s lt-mesos-slave.ERROR mesos-slave.ERROR*
*ln -s lt-mesos-slave.WARNING mesos-slave.WARNING*
*ln -s lt-mesos-slave.INFO mesos-slave.INFO*
I see only these suspicious lines in lt-mesos-master.ERROR
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path
'/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to
canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into an
absolute path
E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log file:
Failed to canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO
into an absolute path
Regards,
Praveenesh
On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler <bm...@twitter.com>wrote:
> 1. You could run it as a daemon, what you're doing with nohup is fine as
> well.
>
> 2. The webui shows various resource statistics: [image: Inline image 2]
>
> *Total memory* indicates the total amount of memory available on all the
> slaves.
> Total Memory should equal Used + Idle + Offered
> *
> *
> *Used:* Memory used by the frameworks.
> *Offered:* This memory has been offered but the framework hasn't
> responded yet.
> *Idle:* Any remaining memory in the system.
>
> 3. Are you running off trunk or a release? What is the command line you're
> running to start the master? Can you provide the master's log output?
>
> On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <pr...@gmail.com>wrote:
>
>> Hi all,
>>
>> I have been able to successfully build and install mesos on my physical
>> nodes. I can see the mesos web-url with list of mesos slaves. However I am
>> finding few difficulties w.r.t to following things:
>>
>> *1. How to start mesos-master.sh and mesos-slave.sh as a background
>> process
>>
>> ? *
>> I am seeing all my output on the console. Since I want to remotely start
>> the service on the cluster, I tried using nohup. Its working for now, but
>> is it the right way? I am hoping there should be better way to do that.
>>
>> *2.* *Whatever memory I am seeing on the mesos-web url, is it the total
>> memory or the free memory available on my system ?*
>>
>> I am seeing very low memory (0.001 GB) -- Does it indicate the current
>> available memory ?
>>
>> 3. I have enable the "log_dir" directory, while starting the mesos-master
>> and slaves. I can see the logs for my slave machines from mesos web url,
>> but not able to see logs for my master. Its saying *"Failed to Initialize
>> .... Retrying"*. I can see work directory is there as well as the logs are
>>
>> also there. Set the permission to 777 also. Still no success.
>>
>>
>> Any help on the above to understand mesos, would be very helpful.
>>
>> Regards,
>> Praveenesh
>>
>
>
Re: Problems in starting mesos
Posted by Benjamin Mahler <bm...@twitter.com>.
1. You could run it as a daemon, what you're doing with nohup is fine as
well.
2. The webui shows various resource statistics: [image: Inline image 2]
*Total memory* indicates the total amount of memory available on all the
slaves.
Total Memory should equal Used + Idle + Offered
*
*
*Used:* Memory used by the frameworks.
*Offered:* This memory has been offered but the framework hasn't responded
yet.
*Idle:* Any remaining memory in the system.
3. Are you running off trunk or a release? What is the command line you're
running to start the master? Can you provide the master's log output?
On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi all,
>
> I have been able to successfully build and install mesos on my physical
> nodes. I can see the mesos web-url with list of mesos slaves. However I am
> finding few difficulties w.r.t to following things:
>
> *1. How to start mesos-master.sh and mesos-slave.sh as a background process
> ? *
> I am seeing all my output on the console. Since I want to remotely start
> the service on the cluster, I tried using nohup. Its working for now, but
> is it the right way? I am hoping there should be better way to do that.
>
> *2.* *Whatever memory I am seeing on the mesos-web url, is it the total
> memory or the free memory available on my system ?*
> I am seeing very low memory (0.001 GB) -- Does it indicate the current
> available memory ?
>
> 3. I have enable the "log_dir" directory, while starting the mesos-master
> and slaves. I can see the logs for my slave machines from mesos web url,
> but not able to see logs for my master. Its saying *"Failed to Initialize
> .... Retrying"*. I can see work directory is there as well as the logs are
> also there. Set the permission to 777 also. Still no success.
>
>
> Any help on the above to understand mesos, would be very helpful.
>
> Regards,
> Praveenesh
>