You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Hao Ren <in...@gmail.com> on 2019/04/03 22:45:02 UTC
zeppelin-daemon doesn't work in docker container
Hi,
I am testing zeppelin in docker container by using the official image:
https://hub.docker.com/r/apache/zeppelin/dockerfile
I notice that the Dockfile above is using "bin/zeppelin.sh" to start
zeppelin, instead of "bin/zeppelin-daemon start".
I am not clear on the difference between the two scripts. So I did some
tests.
What I did:
* remove ENTRYPOINT and CMD from Dockfile
* build and run the container
* Inside the container terminal, use "bin/zeppelin-daemon.sh start" to
start Zeppelin
The problem I met:
* Zeppelin can be launched
* However, when executing "spark.version" in notebook, Zeppelin hangs
* http://localhost:8080/#/interpreter does not show anything
If I stop the daemon and use "bin/zeppelin.sh" to start Zeppelin,
everything works as expected.
Further investigation shows that the two scripts seem to run the same
command, except "bin/zeppelin-daemon start" using "nohup nice" to run the
command in the background. So I tried to run "nohup nice bin/zeppelin.sh",
unsurprisingly, the problem mentioned above appears again.
Once I bring the process to the foreground, the frozen page works again!
Could anyone explain what has happened? I am so confused about this. It
might be related to the way background process works in the docker
container. I am new to docker and Zeppelin.
Any help is highly appreciated.
Thank you.
--
Hao Ren
Software Engineer in Machine Learning @ Criteo
Paris, France
Re: zeppelin-daemon doesn't work in docker container
Posted by Hao Ren <in...@gmail.com>.
Hi Moon,
Sorry for the late reply.
I would like to use bin/zeppelin-daemon.sh, because I want to keep it as
the background process while I can do some other work in the container.
I also tried to use `nohup bin/zeppelin.sh &` command, but still, once the
interpreter process is running in the background, it hangs.
You can easily produce the problem by using the official docker image of
Zeppelin: https://hub.docker.com/r/apache/zeppelin
Step 1: Change the last line of Dockfile to "CMD ["/bin/bash"]"
Step 2: Run "bin/zeppelin-daemon.sh start" in the docker container.
There might be some docker mechanism I am not aware of.
Thank you.
Hao
On Wed, Apr 10, 2019 at 11:37 AM moon soo Lee <le...@gmail.com> wrote:
> Hao Ren,
>
> It looks like other interpreters are working even if Zeppelin is started
> inside the container using "bin/zeppelin-daemon.sh start" command, except
> for Spark interpreter. So I guess it is somehow related to Spark.
>
> One question, do you have any particular reason try to use
> bin/zeppelin-daemon.sh script instead of bin/zeppelin.sh inside the
> container?
>
> Thanks,
> moon
>
> On Thu, Apr 4, 2019 at 11:38 PM Hao Ren <in...@gmail.com> wrote:
>
>> Here is the ps tree for the two cases.
>> And I have no idea why Zeppelin fork some sub processes for STTY when
>> Zeppelin is hanging
>>
>> * Working
>> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>> root 1 0.0 0.0 1144 4 ? Ss 08:24 0:00
>> /dev/init -- /usr/local/bin/bootstrap.sh
>> root 7 0.0 0.0 48696 3132 ? S 08:24 0:00 su -l
>> h.ren
>> h.ren 25 0.0 0.0 20144 3816 ? S 08:25 0:00 \_ -su
>> h.ren 279 0.0 0.0 36144 3272 ? R+ 08:27 0:00 \_
>> ps auxf -G1
>> h.ren 79 6.0 1.5 6305268 520520 ? Sl 08:25 0:10
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
>> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
>> h.ren 133 0.0 0.0 11492 3124 ? S 08:27 0:00 \_
>> /bin/bash /app/zeppelin/bin/interpreter.sh -d
>> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r : -l
>> h.ren 145 0.0 0.0 11492 2112 ? S 08:27 0:00 \_
>> /bin/bash /app/zeppelin/bin/interpreter.sh -d
>> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r :
>> h.ren 146 110 2.0 14354788 670188 ? Sl 08:27 0:18
>> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
>>
>> * Hanging
>> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>> root 1 0.1 0.0 1144 4 ? Ss 12:56 0:00
>> /dev/init -- /usr/local/bin/bootstrap.sh
>> root 7 0.0 0.0 48696 3128 ? S 12:56 0:00 su -l
>> h.ren
>> h.ren 25 0.0 0.0 20152 3832 ? S 12:56 0:00 \_ -su
>> h.ren 208 0.0 0.0 36144 3172 ? R+ 12:57 0:00 \_
>> ps auxf -g1
>> h.ren 79 40.2 1.5 6238704 516152 ? Tl 12:57 0:09
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
>> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
>> h.ren 131 0.0 0.0 11492 3088 ? T 12:57 0:00 \_
>> /bin/bash /app/zeppelin/bin/interpreter.sh -d
>> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r : -l
>> h.ren 143 0.0 0.0 11492 1980 ? T 12:57 0:00 \_
>> /bin/bash /app/zeppelin/bin/interpreter.sh -d
>> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r :
>> h.ren 144 24.3 0.8 12053996 287476 ? Tl 12:57 0:03
>> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
>> h.ren 205 0.0 0.0 4524 720 ? T 12:57 0:00
>> \_ sh -c stty -icanon min 1 -icrnl -inlcr -ixon < /dev/tty
>> h.ren 206 0.0 0.0 6136 708 ? T 12:57 0:00
>> \_ stty -icanon min 1 -icrnl -inlcr -ixon
>>
>> On Thu, Apr 4, 2019 at 12:45 AM Hao Ren <in...@gmail.com> wrote:
>>
>>> Hi,
>>> I am testing zeppelin in docker container by using the official image:
>>> https://hub.docker.com/r/apache/zeppelin/dockerfile
>>>
>>> I notice that the Dockfile above is using "bin/zeppelin.sh" to start
>>> zeppelin, instead of "bin/zeppelin-daemon start".
>>>
>>> I am not clear on the difference between the two scripts. So I did some
>>> tests.
>>>
>>> What I did:
>>> * remove ENTRYPOINT and CMD from Dockfile
>>> * build and run the container
>>> * Inside the container terminal, use "bin/zeppelin-daemon.sh start" to
>>> start Zeppelin
>>>
>>> The problem I met:
>>> * Zeppelin can be launched
>>> * However, when executing "spark.version" in notebook, Zeppelin hangs
>>> * http://localhost:8080/#/interpreter does not show anything
>>>
>>> If I stop the daemon and use "bin/zeppelin.sh" to start Zeppelin,
>>> everything works as expected.
>>>
>>> Further investigation shows that the two scripts seem to run the same
>>> command, except "bin/zeppelin-daemon start" using "nohup nice" to run the
>>> command in the background. So I tried to run "nohup nice bin/zeppelin.sh",
>>> unsurprisingly, the problem mentioned above appears again.
>>>
>>> Once I bring the process to the foreground, the frozen page works again!
>>>
>>> Could anyone explain what has happened? I am so confused about this. It
>>> might be related to the way background process works in the docker
>>> container. I am new to docker and Zeppelin.
>>> Any help is highly appreciated.
>>>
>>> Thank you.
>>>
>>> --
>>> Hao Ren
>>>
>>> Software Engineer in Machine Learning @ Criteo
>>>
>>> Paris, France
>>>
>>
>>
>> --
>> Hao Ren
>>
>> Software Engineer in Machine Learning @ Criteo
>>
>> Paris, France
>>
>
--
Hao Ren
Software Engineer in Machine Learning @ Criteo
Paris, France
Re: zeppelin-daemon doesn't work in docker container
Posted by moon soo Lee <le...@gmail.com>.
Hao Ren,
It looks like other interpreters are working even if Zeppelin is started
inside the container using "bin/zeppelin-daemon.sh start" command, except
for Spark interpreter. So I guess it is somehow related to Spark.
One question, do you have any particular reason try to use
bin/zeppelin-daemon.sh script instead of bin/zeppelin.sh inside the
container?
Thanks,
moon
On Thu, Apr 4, 2019 at 11:38 PM Hao Ren <in...@gmail.com> wrote:
> Here is the ps tree for the two cases.
> And I have no idea why Zeppelin fork some sub processes for STTY when
> Zeppelin is hanging
>
> * Working
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 1 0.0 0.0 1144 4 ? Ss 08:24 0:00 /dev/init
> -- /usr/local/bin/bootstrap.sh
> root 7 0.0 0.0 48696 3132 ? S 08:24 0:00 su -l
> h.ren
> h.ren 25 0.0 0.0 20144 3816 ? S 08:25 0:00 \_ -su
> h.ren 279 0.0 0.0 36144 3272 ? R+ 08:27 0:00 \_
> ps auxf -G1
> h.ren 79 6.0 1.5 6305268 520520 ? Sl 08:25 0:10
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
> h.ren 133 0.0 0.0 11492 3124 ? S 08:27 0:00 \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r : -l
> h.ren 145 0.0 0.0 11492 2112 ? S 08:27 0:00 \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r :
> h.ren 146 110 2.0 14354788 670188 ? Sl 08:27 0:18
> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
>
> * Hanging
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 1 0.1 0.0 1144 4 ? Ss 12:56 0:00 /dev/init
> -- /usr/local/bin/bootstrap.sh
> root 7 0.0 0.0 48696 3128 ? S 12:56 0:00 su -l
> h.ren
> h.ren 25 0.0 0.0 20152 3832 ? S 12:56 0:00 \_ -su
> h.ren 208 0.0 0.0 36144 3172 ? R+ 12:57 0:00 \_
> ps auxf -g1
> h.ren 79 40.2 1.5 6238704 516152 ? Tl 12:57 0:09
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
> h.ren 131 0.0 0.0 11492 3088 ? T 12:57 0:00 \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r : -l
> h.ren 143 0.0 0.0 11492 1980 ? T 12:57 0:00 \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r :
> h.ren 144 24.3 0.8 12053996 287476 ? Tl 12:57 0:03
> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
> h.ren 205 0.0 0.0 4524 720 ? T 12:57 0:00
> \_ sh -c stty -icanon min 1 -icrnl -inlcr -ixon < /dev/tty
> h.ren 206 0.0 0.0 6136 708 ? T 12:57 0:00
> \_ stty -icanon min 1 -icrnl -inlcr -ixon
>
> On Thu, Apr 4, 2019 at 12:45 AM Hao Ren <in...@gmail.com> wrote:
>
>> Hi,
>> I am testing zeppelin in docker container by using the official image:
>> https://hub.docker.com/r/apache/zeppelin/dockerfile
>>
>> I notice that the Dockfile above is using "bin/zeppelin.sh" to start
>> zeppelin, instead of "bin/zeppelin-daemon start".
>>
>> I am not clear on the difference between the two scripts. So I did some
>> tests.
>>
>> What I did:
>> * remove ENTRYPOINT and CMD from Dockfile
>> * build and run the container
>> * Inside the container terminal, use "bin/zeppelin-daemon.sh start" to
>> start Zeppelin
>>
>> The problem I met:
>> * Zeppelin can be launched
>> * However, when executing "spark.version" in notebook, Zeppelin hangs
>> * http://localhost:8080/#/interpreter does not show anything
>>
>> If I stop the daemon and use "bin/zeppelin.sh" to start Zeppelin,
>> everything works as expected.
>>
>> Further investigation shows that the two scripts seem to run the same
>> command, except "bin/zeppelin-daemon start" using "nohup nice" to run the
>> command in the background. So I tried to run "nohup nice bin/zeppelin.sh",
>> unsurprisingly, the problem mentioned above appears again.
>>
>> Once I bring the process to the foreground, the frozen page works again!
>>
>> Could anyone explain what has happened? I am so confused about this. It
>> might be related to the way background process works in the docker
>> container. I am new to docker and Zeppelin.
>> Any help is highly appreciated.
>>
>> Thank you.
>>
>> --
>> Hao Ren
>>
>> Software Engineer in Machine Learning @ Criteo
>>
>> Paris, France
>>
>
>
> --
> Hao Ren
>
> Software Engineer in Machine Learning @ Criteo
>
> Paris, France
>
Re: zeppelin-daemon doesn't work in docker container
Posted by Hao Ren <in...@gmail.com>.
Here is the ps tree for the two cases.
And I have no idea why Zeppelin fork some sub processes for STTY when
Zeppelin is hanging
* Working
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1144 4 ? Ss 08:24 0:00 /dev/init
-- /usr/local/bin/bootstrap.sh
root 7 0.0 0.0 48696 3132 ? S 08:24 0:00 su -l h.ren
h.ren 25 0.0 0.0 20144 3816 ? S 08:25 0:00 \_ -su
h.ren 279 0.0 0.0 36144 3272 ? R+ 08:27 0:00 \_ ps
auxf -G1
h.ren 79 6.0 1.5 6305268 520520 ? Sl 08:25 0:10
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
-Xmx1024m -XX:MaxPermSize=512m -Dlog4j
h.ren 133 0.0 0.0 11492 3124 ? S 08:27 0:00 \_
/bin/bash /app/zeppelin/bin/interpreter.sh -d
/app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r : -l
h.ren 145 0.0 0.0 11492 2112 ? S 08:27 0:00 \_
/bin/bash /app/zeppelin/bin/interpreter.sh -d
/app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r :
h.ren 146 110 2.0 14354788 670188 ? Sl 08:27 0:18
\_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
* Hanging
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 1144 4 ? Ss 12:56 0:00 /dev/init
-- /usr/local/bin/bootstrap.sh
root 7 0.0 0.0 48696 3128 ? S 12:56 0:00 su -l h.ren
h.ren 25 0.0 0.0 20152 3832 ? S 12:56 0:00 \_ -su
h.ren 208 0.0 0.0 36144 3172 ? R+ 12:57 0:00 \_ ps
auxf -g1
h.ren 79 40.2 1.5 6238704 516152 ? Tl 12:57 0:09
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
-Xmx1024m -XX:MaxPermSize=512m -Dlog4j
h.ren 131 0.0 0.0 11492 3088 ? T 12:57 0:00 \_
/bin/bash /app/zeppelin/bin/interpreter.sh -d
/app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r : -l
h.ren 143 0.0 0.0 11492 1980 ? T 12:57 0:00 \_
/bin/bash /app/zeppelin/bin/interpreter.sh -d
/app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r :
h.ren 144 24.3 0.8 12053996 287476 ? Tl 12:57 0:03
\_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
h.ren 205 0.0 0.0 4524 720 ? T 12:57 0:00
\_ sh -c stty -icanon min 1 -icrnl -inlcr -ixon < /dev/tty
h.ren 206 0.0 0.0 6136 708 ? T 12:57 0:00
\_ stty -icanon min 1 -icrnl -inlcr -ixon
On Thu, Apr 4, 2019 at 12:45 AM Hao Ren <in...@gmail.com> wrote:
> Hi,
> I am testing zeppelin in docker container by using the official image:
> https://hub.docker.com/r/apache/zeppelin/dockerfile
>
> I notice that the Dockfile above is using "bin/zeppelin.sh" to start
> zeppelin, instead of "bin/zeppelin-daemon start".
>
> I am not clear on the difference between the two scripts. So I did some
> tests.
>
> What I did:
> * remove ENTRYPOINT and CMD from Dockfile
> * build and run the container
> * Inside the container terminal, use "bin/zeppelin-daemon.sh start" to
> start Zeppelin
>
> The problem I met:
> * Zeppelin can be launched
> * However, when executing "spark.version" in notebook, Zeppelin hangs
> * http://localhost:8080/#/interpreter does not show anything
>
> If I stop the daemon and use "bin/zeppelin.sh" to start Zeppelin,
> everything works as expected.
>
> Further investigation shows that the two scripts seem to run the same
> command, except "bin/zeppelin-daemon start" using "nohup nice" to run the
> command in the background. So I tried to run "nohup nice bin/zeppelin.sh",
> unsurprisingly, the problem mentioned above appears again.
>
> Once I bring the process to the foreground, the frozen page works again!
>
> Could anyone explain what has happened? I am so confused about this. It
> might be related to the way background process works in the docker
> container. I am new to docker and Zeppelin.
> Any help is highly appreciated.
>
> Thank you.
>
> --
> Hao Ren
>
> Software Engineer in Machine Learning @ Criteo
>
> Paris, France
>
--
Hao Ren
Software Engineer in Machine Learning @ Criteo
Paris, France