You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cmda.apache.org by "Pan, Lei (398K)" <le...@jpl.nasa.gov> on 2016/07/19 23:28:50 UTC
auto start not stable
Hi Qihao:
I have burned an EC2 image and am now testing
the auto start of our services. I found it to be unstable.
There are two steps in auto start: (1) the start of the
Docker container when the EC2 instance is launched;
and (2) the start of the CMU and JPL services, especially
the CMDA backend, when the Docker container is up.
The (1) step is still being tested, but at least every time
I reboot the EC2 instance, I get Docker player, though
sometimes it takes a long time.
The (2) is not stable. What I found is that when I manually
run the Docker instance, I do not always see the CMDA
backend started. Sometimes if I log in as svc to the
container, I may trigger the gunicorn process to be started,
other times no matter how many times I log in, I do not
see gunicorn proceses.
You can debug the issue using the following steps:
(1) Get on to the EC2 instance, from which I burned the image:
ssh -i climate-default.pem ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com
(2) stop the running container if any, by
docker ps
and
docker stop <container ID>
(3) run docker using the line found in
/etc/init.d/launch-cmac-docker.sh
which is the script used to automatically launch docker
(4) Point your browser to:
<http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
Go to any cmda service, such as twoDimMap, and try to run it.
(5) If you see “error!”, you can try to log in to the Docker container
ssh -p 8122 svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com
and type:
ps -aelf|grep python
to see if the gunicorn processes are there.
If they are there, then you can go back to your browser
to request cmda services again.
What I have seen is that sometimes after logging in, I
can see the gunicorn processes and the cmda services
are then up, but other times even if I log in many times,
I do not see the gunicorn processes. This is the instability
I mean. This puzzles me because you put in the line “sh start.sh”
in /home/svc/.bashrc, and it should be called every time I log
in as svc, correct?
Before asking you to check on this, I actually tried to use a different
way — upstart — to launch the cmda services. But unlike in the full
linux env, upstart does not seem to work in Docker. The service script
that I put under /etc/init/ did not work.
Thanks,
-Lei
Re: auto start not stable
Posted by "Pan, Lei (398K)" <le...@jpl.nasa.gov>.
Hi Qihao:
Here is the content of
/etc/init.d/launch-cmac-docker.sh
So the line does have “/bin/bash”
at the end. Please give it a try! Thanks, -Lei
!/bin/bash
export HOME="/home/ubuntu"
cd $HOME
. $HOME/.bash_profile
### sudo service docker start
### /bin/su - ubuntu
echo `whoami` > $HOME/mytest.log
### ls > $HOME/mytest.log
exec /usr/bin/docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
echo "from docker run 2" >> $HOME/mytest.log
From: Qihao Bao <qi...@cmu.edu>>
Date: Tuesday, July 19, 2016 at 4:41 PM
To: JPL <le...@jpl.nasa.gov>>
Cc: Jia Zhang <ji...@sv.cmu.edu>>, Seungwon Lee <Se...@jpl.nasa.gov>>, Benyang Tang <be...@jpl.nasa.gov>>, "Zhai, Chengxing (398K)" <ch...@jpl.nasa.gov>>, "dev@cmda.incubator.apache.org<ma...@cmda.incubator.apache.org>" <de...@cmda.incubator.apache.org>>
Subject: Re: auto start not stable
Hi Lei,
What's the command line that you used to start docker image inside of a VM?
I guess that's the issue.
I remembered i used something similar to
++++++++++++++++++++++++++++++++++++++++++
docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
++++++++++++++++++++++++++++++++++++++++++
The /bin/bash is needed to start a bash in the docker, which is necessary for auto start...
Qihao
On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <le...@jpl.nasa.gov>> wrote:
Hi Qihao:
I have burned an EC2 image and am now testing
the auto start of our services. I found it to be unstable.
There are two steps in auto start: (1) the start of the
Docker container when the EC2 instance is launched;
and (2) the start of the CMU and JPL services, especially
the CMDA backend, when the Docker container is up.
The (1) step is still being tested, but at least every time
I reboot the EC2 instance, I get Docker player, though
sometimes it takes a long time.
The (2) is not stable. What I found is that when I manually
run the Docker instance, I do not always see the CMDA
backend started. Sometimes if I log in as svc to the
container, I may trigger the gunicorn process to be started,
other times no matter how many times I log in, I do not
see gunicorn proceses.
You can debug the issue using the following steps:
(1) Get on to the EC2 instance, from which I burned the image:
ssh -i climate-default.pem ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<ma...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
(2) stop the running container if any, by
docker ps
and
docker stop <container ID>
(3) run docker using the line found in
/etc/init.d/launch-cmac-docker.sh
which is the script used to automatically launch docker
(4) Point your browser to:
<http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
Go to any cmda service, such as twoDimMap, and try to run it.
(5) If you see “error!”, you can try to log in to the Docker container
ssh -p 8122 svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<ma...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
and type:
ps -aelf|grep python
to see if the gunicorn processes are there.
If they are there, then you can go back to your browser
to request cmda services again.
What I have seen is that sometimes after logging in, I
can see the gunicorn processes and the cmda services
are then up, but other times even if I log in many times,
I do not see the gunicorn processes. This is the instability
I mean. This puzzles me because you put in the line “sh start.sh”
in /home/svc/.bashrc, and it should be called every time I log
in as svc, correct?
Before asking you to check on this, I actually tried to use a different
way — upstart — to launch the cmda services. But unlike in the full
linux env, upstart does not seem to work in Docker. The service script
that I put under /etc/init/ did not work.
Thanks,
-Lei
Re: auto start not stable
Posted by Qihao Bao <qi...@cmu.edu>.
Lei,
I modified .bashrc in */root* folder.
I deleted 'sudo su svc' and appended 'su - svc -c "cd
/home/svc/new_github/JPL-WebService/JPL_CMDA/services/svc;sh start.sh"'.
Thanks
On Wed, Jul 20, 2016 at 3:16 PM, Pan, Lei (398K) <le...@jpl.nasa.gov>
wrote:
> Hi Qihao:
>
> Thanks a lot!
>
> I will give it a try. Could you please tell me
> what exactly you have changed and where you did
> the change, so I can make sure that I have your
> changes when I burn another EC2 image.
>
> -Lei
>
>
> On 7/20/16, 2:07 PM, "Qihao Bao" <qi...@cmu.edu> wrote:
>
> >Hi Lei,
> >
> >Yes, you are right. I reproduced the issue that JPL-backend failed to auto
> >start.
> >
> >Good news is that i found another way to run auto-start command lines for
> >JPL-Backend, which is use root user to run scripts as another user,
> >instead
> >of switching to another user account.
> >I assume that is more stable for summer school.
> >
> >I also update the docker images.
> >
> >If you have time, please give a try. Restart the VM and see if it works.
> >
> >
> >Thanks.
> >
> >
> >
> >Br.
> >Qihao
> >
> >On Wed, Jul 20, 2016 at 1:17 AM, Pan, Lei (398K) <le...@jpl.nasa.gov>
> >wrote:
> >
> >> Hi Qihao:
> >>
> >> Here is the content of
> >> /etc/init.d/launch-cmac-docker.sh
> >> So the line does have ³/bin/bash²
> >> at the end. Please give it a try! Thanks, -Lei
> >>
> >>
> >> !/bin/bash
> >>
> >> export HOME="/home/ubuntu"
> >>
> >> cd $HOME
> >>
> >> . $HOME/.bash_profile
> >>
> >> ### sudo service docker start
> >>
> >> ### /bin/su - ubuntu
> >>
> >> echo `whoami` > $HOME/mytest.log
> >>
> >> ### ls > $HOME/mytest.log
> >>
> >> exec /usr/bin/docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p
> >> 8180:8880 -p 8090:8890 -p 8023:8023 -v
> >>/mnt/xvdf/data_2015:/mnt/data_2015
> >> -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
> >>
> >> echo "from docker run 2" >> $HOME/mytest.log
> >>
> >>
> >>
> >>
> >> From: Qihao Bao <qi...@cmu.edu>>
> >> Date: Tuesday, July 19, 2016 at 4:41 PM
> >> To: JPL <le...@jpl.nasa.gov>>
> >> Cc: Jia Zhang <ji...@sv.cmu.edu>>,
> >> Seungwon Lee
> >><Se...@jpl.nasa.gov>>,
> >> Benyang Tang
> >><be...@jpl.nasa.gov>>,
> >> "Zhai, Chengxing (398K)" <chengxing.zhai@jpl.nasa.gov<mailto:
> >> chengxing.zhai@jpl.nasa.gov>>, "dev@cmda.incubator.apache.org<mailto:
> >> dev@cmda.incubator.apache.org>" <dev@cmda.incubator.apache.org<mailto:
> >> dev@cmda.incubator.apache.org>>
> >> Subject: Re: auto start not stable
> >>
> >> Hi Lei,
> >>
> >> What's the command line that you used to start docker image inside of a
> >>VM?
> >> I guess that's the issue.
> >>
> >> I remembered i used something similar to
> >> ++++++++++++++++++++++++++++++++++++++++++
> >> docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p
> >> 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i
> >> cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
> >> ++++++++++++++++++++++++++++++++++++++++++
> >>
> >> The /bin/bash is needed to start a bash in the docker, which is
> >>necessary
> >> for auto start...
> >>
> >> Qihao
> >>
> >> On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <lei.pan@jpl.nasa.gov
> >> <ma...@jpl.nasa.gov>> wrote:
> >> Hi Qihao:
> >>
> >> I have burned an EC2 image and am now testing
> >> the auto start of our services. I found it to be unstable.
> >>
> >> There are two steps in auto start: (1) the start of the
> >> Docker container when the EC2 instance is launched;
> >> and (2) the start of the CMU and JPL services, especially
> >> the CMDA backend, when the Docker container is up.
> >>
> >> The (1) step is still being tested, but at least every time
> >> I reboot the EC2 instance, I get Docker player, though
> >> sometimes it takes a long time.
> >>
> >> The (2) is not stable. What I found is that when I manually
> >> run the Docker instance, I do not always see the CMDA
> >> backend started. Sometimes if I log in as svc to the
> >> container, I may trigger the gunicorn process to be started,
> >> other times no matter how many times I log in, I do not
> >> see gunicorn proceses.
> >>
> >> You can debug the issue using the following steps:
> >> (1) Get on to the EC2 instance, from which I burned the image:
> >> ssh -i climate-default.pem
> >> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> >> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
> >> (2) stop the running container if any, by
> >> docker ps
> >> and
> >> docker stop <container ID>
> >> (3) run docker using the line found in
> >>
> >> /etc/init.d/launch-cmac-docker.sh
> >>
> >> which is the script used to automatically launch docker
> >>
> >> (4) Point your browser to:
> >>
> >> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> >> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> >> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
> >>
> >> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
> >>
> >> Go to any cmda service, such as twoDimMap, and try to run it.
> >>
> >> (5) If you see ³error!², you can try to log in to the Docker container
> >>
> >> ssh -p 8122
> >>svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> >> svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
> >>
> >> and type:
> >>
> >> ps -aelf|grep python
> >>
> >> to see if the gunicorn processes are there.
> >>
> >> If they are there, then you can go back to your browser
> >>
> >> to request cmda services again.
> >>
> >>
> >> What I have seen is that sometimes after logging in, I
> >>
> >> can see the gunicorn processes and the cmda services
> >>
> >> are then up, but other times even if I log in many times,
> >>
> >> I do not see the gunicorn processes. This is the instability
> >>
> >> I mean. This puzzles me because you put in the line ³sh start.sh²
> >>
> >> in /home/svc/.bashrc, and it should be called every time I log
> >>
> >> in as svc, correct?
> >>
> >>
> >> Before asking you to check on this, I actually tried to use a different
> >>
> >> way ‹ upstart ‹ to launch the cmda services. But unlike in the full
> >>
> >> linux env, upstart does not seem to work in Docker. The service script
> >>
> >> that I put under /etc/init/ did not work.
> >>
> >>
> >> Thanks,
> >>
> >> -Lei
> >>
> >>
> >>
> >>
> >>
> >>
>
>
Re: auto start not stable
Posted by "Pan, Lei (398K)" <le...@jpl.nasa.gov>.
Hi Qihao:
Thanks a lot!
I will give it a try. Could you please tell me
what exactly you have changed and where you did
the change, so I can make sure that I have your
changes when I burn another EC2 image.
-Lei
On 7/20/16, 2:07 PM, "Qihao Bao" <qi...@cmu.edu> wrote:
>Hi Lei,
>
>Yes, you are right. I reproduced the issue that JPL-backend failed to auto
>start.
>
>Good news is that i found another way to run auto-start command lines for
>JPL-Backend, which is use root user to run scripts as another user,
>instead
>of switching to another user account.
>I assume that is more stable for summer school.
>
>I also update the docker images.
>
>If you have time, please give a try. Restart the VM and see if it works.
>
>
>Thanks.
>
>
>
>Br.
>Qihao
>
>On Wed, Jul 20, 2016 at 1:17 AM, Pan, Lei (398K) <le...@jpl.nasa.gov>
>wrote:
>
>> Hi Qihao:
>>
>> Here is the content of
>> /etc/init.d/launch-cmac-docker.sh
>> So the line does have ³/bin/bash²
>> at the end. Please give it a try! Thanks, -Lei
>>
>>
>> !/bin/bash
>>
>> export HOME="/home/ubuntu"
>>
>> cd $HOME
>>
>> . $HOME/.bash_profile
>>
>> ### sudo service docker start
>>
>> ### /bin/su - ubuntu
>>
>> echo `whoami` > $HOME/mytest.log
>>
>> ### ls > $HOME/mytest.log
>>
>> exec /usr/bin/docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p
>> 8180:8880 -p 8090:8890 -p 8023:8023 -v
>>/mnt/xvdf/data_2015:/mnt/data_2015
>> -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
>>
>> echo "from docker run 2" >> $HOME/mytest.log
>>
>>
>>
>>
>> From: Qihao Bao <qi...@cmu.edu>>
>> Date: Tuesday, July 19, 2016 at 4:41 PM
>> To: JPL <le...@jpl.nasa.gov>>
>> Cc: Jia Zhang <ji...@sv.cmu.edu>>,
>> Seungwon Lee
>><Se...@jpl.nasa.gov>>,
>> Benyang Tang
>><be...@jpl.nasa.gov>>,
>> "Zhai, Chengxing (398K)" <chengxing.zhai@jpl.nasa.gov<mailto:
>> chengxing.zhai@jpl.nasa.gov>>, "dev@cmda.incubator.apache.org<mailto:
>> dev@cmda.incubator.apache.org>" <dev@cmda.incubator.apache.org<mailto:
>> dev@cmda.incubator.apache.org>>
>> Subject: Re: auto start not stable
>>
>> Hi Lei,
>>
>> What's the command line that you used to start docker image inside of a
>>VM?
>> I guess that's the issue.
>>
>> I remembered i used something similar to
>> ++++++++++++++++++++++++++++++++++++++++++
>> docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p
>> 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i
>> cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
>> ++++++++++++++++++++++++++++++++++++++++++
>>
>> The /bin/bash is needed to start a bash in the docker, which is
>>necessary
>> for auto start...
>>
>> Qihao
>>
>> On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <lei.pan@jpl.nasa.gov
>> <ma...@jpl.nasa.gov>> wrote:
>> Hi Qihao:
>>
>> I have burned an EC2 image and am now testing
>> the auto start of our services. I found it to be unstable.
>>
>> There are two steps in auto start: (1) the start of the
>> Docker container when the EC2 instance is launched;
>> and (2) the start of the CMU and JPL services, especially
>> the CMDA backend, when the Docker container is up.
>>
>> The (1) step is still being tested, but at least every time
>> I reboot the EC2 instance, I get Docker player, though
>> sometimes it takes a long time.
>>
>> The (2) is not stable. What I found is that when I manually
>> run the Docker instance, I do not always see the CMDA
>> backend started. Sometimes if I log in as svc to the
>> container, I may trigger the gunicorn process to be started,
>> other times no matter how many times I log in, I do not
>> see gunicorn proceses.
>>
>> You can debug the issue using the following steps:
>> (1) Get on to the EC2 instance, from which I burned the image:
>> ssh -i climate-default.pem
>> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
>> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
>> (2) stop the running container if any, by
>> docker ps
>> and
>> docker stop <container ID>
>> (3) run docker using the line found in
>>
>> /etc/init.d/launch-cmac-docker.sh
>>
>> which is the script used to automatically launch docker
>>
>> (4) Point your browser to:
>>
>> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
>> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
>> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
>>
>> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
>>
>> Go to any cmda service, such as twoDimMap, and try to run it.
>>
>> (5) If you see ³error!², you can try to log in to the Docker container
>>
>> ssh -p 8122
>>svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
>> svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
>>
>> and type:
>>
>> ps -aelf|grep python
>>
>> to see if the gunicorn processes are there.
>>
>> If they are there, then you can go back to your browser
>>
>> to request cmda services again.
>>
>>
>> What I have seen is that sometimes after logging in, I
>>
>> can see the gunicorn processes and the cmda services
>>
>> are then up, but other times even if I log in many times,
>>
>> I do not see the gunicorn processes. This is the instability
>>
>> I mean. This puzzles me because you put in the line ³sh start.sh²
>>
>> in /home/svc/.bashrc, and it should be called every time I log
>>
>> in as svc, correct?
>>
>>
>> Before asking you to check on this, I actually tried to use a different
>>
>> way ‹ upstart ‹ to launch the cmda services. But unlike in the full
>>
>> linux env, upstart does not seem to work in Docker. The service script
>>
>> that I put under /etc/init/ did not work.
>>
>>
>> Thanks,
>>
>> -Lei
>>
>>
>>
>>
>>
>>
Re: auto start not stable
Posted by Qihao Bao <qi...@cmu.edu>.
Hi Lei,
Yes, you are right. I reproduced the issue that JPL-backend failed to auto
start.
Good news is that i found another way to run auto-start command lines for
JPL-Backend, which is use root user to run scripts as another user, instead
of switching to another user account.
I assume that is more stable for summer school.
I also update the docker images.
If you have time, please give a try. Restart the VM and see if it works.
Thanks.
Br.
Qihao
On Wed, Jul 20, 2016 at 1:17 AM, Pan, Lei (398K) <le...@jpl.nasa.gov>
wrote:
> Hi Qihao:
>
> Here is the content of
> /etc/init.d/launch-cmac-docker.sh
> So the line does have “/bin/bash”
> at the end. Please give it a try! Thanks, -Lei
>
>
> !/bin/bash
>
> export HOME="/home/ubuntu"
>
> cd $HOME
>
> . $HOME/.bash_profile
>
> ### sudo service docker start
>
> ### /bin/su - ubuntu
>
> echo `whoami` > $HOME/mytest.log
>
> ### ls > $HOME/mytest.log
>
> exec /usr/bin/docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p
> 8180:8880 -p 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015
> -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
>
> echo "from docker run 2" >> $HOME/mytest.log
>
>
>
>
> From: Qihao Bao <qi...@cmu.edu>>
> Date: Tuesday, July 19, 2016 at 4:41 PM
> To: JPL <le...@jpl.nasa.gov>>
> Cc: Jia Zhang <ji...@sv.cmu.edu>>,
> Seungwon Lee <Se...@jpl.nasa.gov>>,
> Benyang Tang <be...@jpl.nasa.gov>>,
> "Zhai, Chengxing (398K)" <chengxing.zhai@jpl.nasa.gov<mailto:
> chengxing.zhai@jpl.nasa.gov>>, "dev@cmda.incubator.apache.org<mailto:
> dev@cmda.incubator.apache.org>" <dev@cmda.incubator.apache.org<mailto:
> dev@cmda.incubator.apache.org>>
> Subject: Re: auto start not stable
>
> Hi Lei,
>
> What's the command line that you used to start docker image inside of a VM?
> I guess that's the issue.
>
> I remembered i used something similar to
> ++++++++++++++++++++++++++++++++++++++++++
> docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p
> 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i
> cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
> ++++++++++++++++++++++++++++++++++++++++++
>
> The /bin/bash is needed to start a bash in the docker, which is necessary
> for auto start...
>
> Qihao
>
> On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <lei.pan@jpl.nasa.gov
> <ma...@jpl.nasa.gov>> wrote:
> Hi Qihao:
>
> I have burned an EC2 image and am now testing
> the auto start of our services. I found it to be unstable.
>
> There are two steps in auto start: (1) the start of the
> Docker container when the EC2 instance is launched;
> and (2) the start of the CMU and JPL services, especially
> the CMDA backend, when the Docker container is up.
>
> The (1) step is still being tested, but at least every time
> I reboot the EC2 instance, I get Docker player, though
> sometimes it takes a long time.
>
> The (2) is not stable. What I found is that when I manually
> run the Docker instance, I do not always see the CMDA
> backend started. Sometimes if I log in as svc to the
> container, I may trigger the gunicorn process to be started,
> other times no matter how many times I log in, I do not
> see gunicorn proceses.
>
> You can debug the issue using the following steps:
> (1) Get on to the EC2 instance, from which I burned the image:
> ssh -i climate-default.pem
> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
> (2) stop the running container if any, by
> docker ps
> and
> docker stop <container ID>
> (3) run docker using the line found in
>
> /etc/init.d/launch-cmac-docker.sh
>
> which is the script used to automatically launch docker
>
> (4) Point your browser to:
>
> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
>
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
>
> Go to any cmda service, such as twoDimMap, and try to run it.
>
> (5) If you see “error!”, you can try to log in to the Docker container
>
> ssh -p 8122 svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
>
> and type:
>
> ps -aelf|grep python
>
> to see if the gunicorn processes are there.
>
> If they are there, then you can go back to your browser
>
> to request cmda services again.
>
>
> What I have seen is that sometimes after logging in, I
>
> can see the gunicorn processes and the cmda services
>
> are then up, but other times even if I log in many times,
>
> I do not see the gunicorn processes. This is the instability
>
> I mean. This puzzles me because you put in the line “sh start.sh”
>
> in /home/svc/.bashrc, and it should be called every time I log
>
> in as svc, correct?
>
>
> Before asking you to check on this, I actually tried to use a different
>
> way — upstart — to launch the cmda services. But unlike in the full
>
> linux env, upstart does not seem to work in Docker. The service script
>
> that I put under /etc/init/ did not work.
>
>
> Thanks,
>
> -Lei
>
>
>
>
>
>
Re: auto start not stable
Posted by Qihao Bao <qi...@cmu.edu>.
Hi Lei,
What's the command line that you used to start docker image inside of a VM?
I guess that's the issue.
I remembered i used something similar to
++++++++++++++++++++++++++++++++++++++++++
docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p
8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015 -i
cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
++++++++++++++++++++++++++++++++++++++++++
The /bin/bash is needed to start a bash in the docker, which is necessary
for auto start...
Qihao
On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <le...@jpl.nasa.gov>
wrote:
> Hi Qihao:
>
> I have burned an EC2 image and am now testing
> the auto start of our services. I found it to be unstable.
>
> There are two steps in auto start: (1) the start of the
> Docker container when the EC2 instance is launched;
> and (2) the start of the CMU and JPL services, especially
> the CMDA backend, when the Docker container is up.
>
> The (1) step is still being tested, but at least every time
> I reboot the EC2 instance, I get Docker player, though
> sometimes it takes a long time.
>
> The (2) is not stable. What I found is that when I manually
> run the Docker instance, I do not always see the CMDA
> backend started. Sometimes if I log in as svc to the
> container, I may trigger the gunicorn process to be started,
> other times no matter how many times I log in, I do not
> see gunicorn proceses.
>
> You can debug the issue using the following steps:
> (1) Get on to the EC2 instance, from which I burned the image:
> ssh -i climate-default.pem
> ubuntu@ec2-52-53-198-227.us-west-1.compute.amazonaws.com
> (2) stop the running container if any, by
> docker ps
> and
> docker stop <container ID>
> (3) run docker using the line found in
>
> /etc/init.d/launch-cmac-docker.sh
>
> which is the script used to automatically launch docker
>
> (4) Point your browser to:
>
> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
>
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
>
> Go to any cmda service, such as twoDimMap, and try to run it.
>
> (5) If you see “error!”, you can try to log in to the Docker container
>
> ssh -p 8122 svc@ec2-52-53-198-227.us-west-1.compute.amazonaws.com
>
> and type:
>
> ps -aelf|grep python
>
> to see if the gunicorn processes are there.
>
> If they are there, then you can go back to your browser
>
> to request cmda services again.
>
>
> What I have seen is that sometimes after logging in, I
>
> can see the gunicorn processes and the cmda services
>
> are then up, but other times even if I log in many times,
>
> I do not see the gunicorn processes. This is the instability
>
> I mean. This puzzles me because you put in the line “sh start.sh”
>
> in /home/svc/.bashrc, and it should be called every time I log
>
> in as svc, correct?
>
>
> Before asking you to check on this, I actually tried to use a different
>
> way — upstart — to launch the cmda services. But unlike in the full
>
> linux env, upstart does not seem to work in Docker. The service script
>
> that I put under /etc/init/ did not work.
>
>
> Thanks,
>
> -Lei
>
>
>
>
>
>