You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Andreas Weise <an...@gmail.com> on 2020/07/16 08:51:39 UTC

Deadlock in Zeppelin 0.8.2

Hi, today we faced a deadlock in our 0.8.2 production environment. Find the
thread dump attached.
Found one Java-level deadlock:
=============================
"qtp459857341-184038":
  waiting to lock monitor 0x00007f0c58018238 (object 0x00000003f3079738, a
java.util.ArrayList),
  which is held by "qtp459857341-183751"
"qtp459857341-183751":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a
java.util.ArrayList),
  which is held by "qtp459857341-183682"
"qtp459857341-183682":
  waiting to lock monitor 0x00007f0c2c19d348 (object 0x00000005d7d54268, a
java.util.LinkedList),
  which is held by "qtp459857341-183610"
"qtp459857341-183610":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a
java.util.ArrayList),
  which is held by "qtp459857341-183682"

Java stack information for the threads listed above:
===================================================
"qtp459857341-184038":
        at org.apache.zeppelin.notebook.Note.getParagraph(Note.java:572)
        - waiting to lock <0x00000003f3079738> (a java.util.ArrayList)
        at
org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:117)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

....
"qtp459857341-183610":
        at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:750)
        - waiting to lock <0x0000000402537780> (a java.util.ArrayList)
        at
org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:872)
        at
org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo(NotebookServer.java:568)
        at
org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove(NotebookServer.java:2194)
        at
org.apache.zeppelin.notebook.Notebook.onParagraphRemove(Notebook.java:1084)
        at org.apache.zeppelin.notebook.Note.removeParagraph(Note.java:446)
        - locked <0x00000005d7d54268> (a java.util.LinkedList)
        at
org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph(NotebookRestApi.java:620)
        at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
...

Please let me know, if that is already fixed in > 0.8.2 or requires a Jira
Ticket?

Best regards
Andreas

Re: Deadlock in Zeppelin 0.8.2

Posted by Jeff Zhang <zj...@gmail.com>.
I think this fixed in 0.9

Andreas Weise <an...@gmail.com> 于2020年7月16日周四 下午4:52写道：

> Hi, today we faced a deadlock in our 0.8.2 production environment. Find
> the thread dump attached.
> Found one Java-level deadlock:
> =============================
> "qtp459857341-184038":
>   waiting to lock monitor 0x00007f0c58018238 (object 0x00000003f3079738, a
> java.util.ArrayList),
>   which is held by "qtp459857341-183751"
> "qtp459857341-183751":
>   waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a
> java.util.ArrayList),
>   which is held by "qtp459857341-183682"
> "qtp459857341-183682":
>   waiting to lock monitor 0x00007f0c2c19d348 (object 0x00000005d7d54268, a
> java.util.LinkedList),
>   which is held by "qtp459857341-183610"
> "qtp459857341-183610":
>   waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a
> java.util.ArrayList),
>   which is held by "qtp459857341-183682"
>
> Java stack information for the threads listed above:
> ===================================================
> "qtp459857341-184038":
>         at org.apache.zeppelin.notebook.Note.getParagraph(Note.java:572)
>         - waiting to lock <0x00000003f3079738> (a java.util.ArrayList)
>         at
> org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:117)
>         at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>
> ....
> "qtp459857341-183610":
>         at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:750)
>         - waiting to lock <0x0000000402537780> (a java.util.ArrayList)
>         at
> org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:872)
>         at
> org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo(NotebookServer.java:568)
>         at
> org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove(NotebookServer.java:2194)
>         at
> org.apache.zeppelin.notebook.Notebook.onParagraphRemove(Notebook.java:1084)
>         at org.apache.zeppelin.notebook.Note.removeParagraph(Note.java:446)
>         - locked <0x00000005d7d54268> (a java.util.LinkedList)
>         at
> org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph(NotebookRestApi.java:620)
>         at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
> ...
>
> Please let me know, if that is already fixed in > 0.8.2 or requires a Jira
> Ticket?
>
> Best regards
> Andreas
>


-- 
Best Regards

Jeff Zhang