Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[sv <sv...@gmail.com>]

Hi, to clarify:
1) Thin client works fine
2) Thick client using port 11211 works fine
3) The "Client Node" connection is the one that was not working.

I got the Client Node connection (#3) to work by specifying the cache name
in the connection string.  Now it connects, and all SQL works well.  

I thought that this was fixed already in 2.6, so that you didn't have to
specify the cache name anymore?  In any case, it works.  

The next problem that I ran into:  executing scala code (using %ignite)
makes the entire Ignite interpreter in Zeppelin hang, without any errors in
logs.  How do I debug this?  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[sv <sv...@gmail.com>]

Evgenii,

"Client node didn't even join the topology. Most possible you have incorrect
discovery configuration. Also, I'd recommend to check client logs for more
information why node can't connect to the cluster."

In Zeppelin setup for the Ignite interpreter, it uses the same configuration
for Scala connection as it does for Ignite SQL connections.  I have tested
the SQL connections, and they work perfectly (both the 11211 port and the
Node Client using XML).  It is odd that the Scala connection doesn't work.

As far as client logs are concerned, Zeppelin only generates the interpreter
logs, such as, zeppelin-interpreter-ignite--zeppelin-db54cc4b4-kppw2.log,  
and that log only states that the job was started:

 INFO [2018-11-03 09:15:41,893] ({pool-2-thread-5}
SchedulerFactory.java[jobStarted]:109) - Job 20181102-071721_1550658378
started by scheduler org.apache.zeppelin.ignite.IgniteInterpreter113609227

And then nothing after that, as the interpreter process hangs.  

Do you have any idea how to turn on DEBUG level logging for the Zeppelin
Ignite interpreter?  





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[Evgenii Zhuravlev <e....@gmail.com>]

>1) How do I enable DEBUG logging on the Ignite cluster?
It depends on the logger that you use. You have to change the config file
and give to the ignite package DEBUG level. It could be JUL, log4j or
log4j2 config files.

> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology
Client node didn't even join the topology. Most possible you have incorrect
discovery configuration. Also, I'd recommend to check client logs for more
information why node can't connect to the cluster.


Evgenii

вс, 4 нояб. 2018 г. в 8:09, sv <sv...@gmail.com>:

> Evgenii,
>
> 1) How do I enable DEBUG logging on the Ignite cluster?
>
> 2) I used jstack to produce a thread dump of the Zeppelin Ignite
> interpreter
> (which runs in a separate process).  I pasted it below.  Does anything look
> out of place to you?
>
> 2018-11-03 09:32:37
> Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):
>
> "Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f202c001000
> nid=0x37b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "tcp-client-disco-msg-worker-#4" #81 prio=5 os_prio=0
> tid=0x00007f1fd1f2a000
> nid=0x2fb waiting on condition [0x00007f1fce1e6000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:522)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1561)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>    Locked ownable synchronizers:
>         - None
>
> "tcp-client-disco-sock-reader-#3" #80 prio=5 os_prio=0
> tid=0x00007f1fd1f25000 nid=0x2fa in Object.wait() [0x00007f1fce2e7000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1006)
>         - locked <0x00000000ee2bc268> (a java.lang.Object)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>    Locked ownable synchronizers:
>         - None
>
> "tcp-client-disco-sock-writer-#2" #79 prio=5 os_prio=0
> tid=0x00007f1fd1f24800 nid=0x2f9 in Object.wait() [0x00007f1fce3e8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1209)
>         - locked <0x00000000ee2bbd88> (a java.lang.Object)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>    Locked ownable synchronizers:
>         - None
>
> "TcpDiscoverySpi.timer" #78 prio=5 os_prio=0 tid=0x00007f1fd1f20800
> nid=0x2f8 in Object.wait() [0x00007f1fce4e9000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at java.util.TimerThread.mainLoop(Timer.java:526)
>         - locked <0x00000000ee228468> (a java.util.TaskQueue)
>         at java.util.TimerThread.run(Timer.java:505)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-data-loader-flusher-#39" #77 prio=5 os_prio=0 tid=0x00007f1fd1efb000
> nid=0x2f7 waiting on condition [0x00007f1fce5ea000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000edc52cc0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>         at
>
> org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-client-listener-3-#37" #75 prio=5 os_prio=0
> tid=0x00007f1fd1eea800 nid=0x2f6 runnable [0x00007f1fce8eb000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000edb0fdf0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000ed992ff0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000ed992ed8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-client-listener-2-#36" #74 prio=5 os_prio=0
> tid=0x00007f1fd1ee8800 nid=0x2f5 runnable [0x00007f1fce9ec000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000ed98e750> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000ed97e4f8> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000ed97e3e0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-client-listener-1-#35" #73 prio=5 os_prio=0
> tid=0x00007f1fd1ee7000 nid=0x2f4 runnable [0x00007f1fceaed000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000ed979c58> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000ed969a00> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000ed9698e8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-client-listener-0-#34" #72 prio=5 os_prio=0
> tid=0x00007f1fd1e67800 nid=0x2f3 runnable [0x00007f1fcebee000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000ed965160> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000ed954f08> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000ed954df0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "nio-acceptor-#33" #71 prio=5 os_prio=0 tid=0x00007f1fd1e66800 nid=0x2f2
> runnable [0x00007f1fcecef000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000ed944218> (a sun.nio.ch.Util$3)
>         - locked <0x00000000ed944208> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000ed9440f0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007f1fd1cdf800 nid=0x2f1
> waiting on condition [0x00007f1fcf1f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000fd641060> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at
>
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4207)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-comm-3-#32" #66 prio=5 os_prio=0
> tid=0x00007f1fd1cdd800
> nid=0x2f0 runnable [0x00007f1fcf2f1000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000fda143b0> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000fda16430> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000fda14308> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-comm-2-#31" #65 prio=5 os_prio=0
> tid=0x00007f1fd1cd7000
> nid=0x2ef runnable [0x00007f1fcf6f5000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000fda16b38> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000fda49580> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000fda16a90> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-comm-1-#30" #64 prio=5 os_prio=0
> tid=0x00007f1fd1cd8800
> nid=0x2ee runnable [0x00007f1fcf4f3000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000fda49c88> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000fda4bd08> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000fda49be0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-comm-0-#29" #63 prio=5 os_prio=0
> tid=0x00007f1fd1cd5800
> nid=0x2ed runnable [0x00007f1fcf5f4000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000fda4c410> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000fda4e490> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000fda4c368> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "nio-acceptor-#28" #62 prio=5 os_prio=0 tid=0x00007f1fd1cdb000 nid=0x2ec
> runnable [0x00007f1fcf3f2000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000fda13c60> (a sun.nio.ch.Util$3)
>         - locked <0x00000000fda13c50> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000fda13838> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-timeout-worker-#27" #61 prio=5 os_prio=0 tid=0x00007f1fd1c4b800
> nid=0x2e8 in Object.wait() [0x00007f1fcf7f6000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
>
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:203)
>         - locked <0x00000000fd9add58> (a java.lang.Object)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007f1fd1c3f000
> nid=0x2e7
> sleeping[0x00007f1fcf8f7000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3308)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "jvm-pause-detector-worker" #53 daemon prio=5 os_prio=0
> tid=0x00007f1fd184e800 nid=0x2e0 sleeping[0x00007f1fcfffe000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
>
> org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-7-#20" #52 prio=5 os_prio=0 tid=0x00007f1fd1805800
> nid=0x2df waiting on condition [0x00007f1ff836b000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-6-#19" #51 prio=5 os_prio=0 tid=0x00007f1fd1803800
> nid=0x2de waiting on condition [0x00007f1ff846c000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-5-#18" #50 prio=5 os_prio=0 tid=0x00007f1fd1802000
> nid=0x2dd waiting on condition [0x00007f1ff856d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-4-#17" #49 prio=5 os_prio=0 tid=0x00007f1fd1800000
> nid=0x2dc waiting on condition [0x00007f1ff866e000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-3-#16" #48 prio=5 os_prio=0 tid=0x00007f1fd17fe000
> nid=0x2db waiting on condition [0x00007f1ff876f000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-2-#15" #47 prio=5 os_prio=0 tid=0x00007f1fd17fc000
> nid=0x2da waiting on condition [0x00007f1ff8870000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-1-#14" #46 prio=5 os_prio=0 tid=0x00007f1fd17fa800
> nid=0x2d9 waiting on condition [0x00007f1ff8971000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "data-streamer-stripe-0-#13" #45 prio=5 os_prio=0 tid=0x00007f1fd17f8800
> nid=0x2d8 waiting on condition [0x00007f1ff8a72000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-7-#12" #44 prio=5 os_prio=0 tid=0x00007f1fd17f6800 nid=0x2d7
> waiting on condition [0x00007f1ff8b73000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-6-#11" #43 prio=5 os_prio=0 tid=0x00007f1fd17f4800 nid=0x2d6
> waiting on condition [0x00007f1ff8c74000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-5-#10" #42 prio=5 os_prio=0 tid=0x00007f1fd17f2800 nid=0x2d5
> waiting on condition [0x00007f1ff8d75000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-4-#9" #41 prio=5 os_prio=0 tid=0x00007f1fd17f0800 nid=0x2d4
> waiting on condition [0x00007f1ff8e76000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-3-#8" #40 prio=5 os_prio=0 tid=0x00007f1fd17ee800 nid=0x2d3
> waiting on condition [0x00007f1ff8f77000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-2-#7" #39 prio=5 os_prio=0 tid=0x00007f1fd17d3000 nid=0x2d2
> waiting on condition [0x00007f1ff9078000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-1-#6" #38 prio=5 os_prio=0 tid=0x00007f1fd17d1800 nid=0x2d1
> waiting on condition [0x00007f1ff9179000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "sys-stripe-0-#5" #37 prio=5 os_prio=0 tid=0x00007f1fd17cf000 nid=0x2d0
> waiting on condition [0x00007f1ffa0f1000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
>         at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "JobProgressPoller, jobId=20181102-071721_1550658378" #36 prio=5 os_prio=0
> tid=0x00007f1fd0003000 nid=0x2cf waiting on condition [0x00007f1ffadf2000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
>
> org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:61)
>
>    Locked ownable synchronizers:
>         - None
>
> "pool-2-thread-7" #35 prio=5 os_prio=0 tid=0x00007f2008001000 nid=0x2ce
> waiting on condition [0x00007f1ffaef3000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c0006268> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "pool-2-thread-6" #33 prio=5 os_prio=0 tid=0x00007f1ffc005800 nid=0x2c9
> waiting on condition [0x00007f1ffb1f4000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c0006268> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "pool-2-thread-5" #31 prio=5 os_prio=0 tid=0x00007f1ffc004800 nid=0x2c4
> waiting on condition [0x00007f1ffb2f4000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000ee2280f0> (a
> java.util.concurrent.CountDownLatch$Sync)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>         at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:271)
>         at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
>         at
>
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>         at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
>         at
>
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
>         at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
>         - locked <0x00000000fd9eaa10> (a
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
>         at
> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
>         at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)
>         at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
>         at org.apache.ignite.Ignition.start(Ignition.java:327)
>         at
>
> org.apache.zeppelin.ignite.IgniteInterpreter.getIgnite(IgniteInterpreter.java:190)
>         at
>
> org.apache.zeppelin.ignite.IgniteInterpreter.initIgnite(IgniteInterpreter.java:208)
>         at
>
> org.apache.zeppelin.ignite.IgniteInterpreter.open(IgniteInterpreter.java:127)
>         at
>
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>         - locked <0x00000000c000f738> (a
> org.apache.zeppelin.ignite.IgniteInterpreter)
>         - locked <0x00000000c000f810> (a
> org.apache.zeppelin.interpreter.LazyOpenInterpreter)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617)
>         at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>         at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000fda11fc8> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>         - <0x00000000fda12080> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>         - <0x00000000fda12138> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>         - <0x00000000fda121f0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>         - <0x00000000c000f900> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "pool-2-thread-4" #29 prio=5 os_prio=0 tid=0x00007f1ffc003000 nid=0x2bf
> waiting on condition [0x00007f1ffb3f6000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c0006268> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-client-exec-3-5" #28 daemon prio=5 os_prio=0 tid=0x00007f1fd0001000
> nid=0x2bb waiting on condition [0x00007f1ffb4f7000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c001bdb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-client-exec-3-4" #27 daemon prio=5 os_prio=0 tid=0x00007f1fe4001000
> nid=0x2ba waiting on condition [0x00007f2018230000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c001bdb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-client-exec-3-3" #26 daemon prio=5 os_prio=0 tid=0x00007f20001c9000
> nid=0x2b9 waiting on condition [0x00007f1ffb5f8000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c001bdb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "a65ff0b2-c2ba-4d1c-8d67-25570203a476-topology-update" #25 daemon prio=5
> os_prio=0 tid=0x00007f20001c5800 nid=0x2b8 waiting on condition
> [0x00007f1ffb6f9000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
>
> org.apache.ignite.internal.client.impl.GridClientImpl$TopologyUpdaterThread.run(GridClientImpl.java:521)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-client-top-lsnr-1-2" #24 daemon prio=5 os_prio=0
> tid=0x00007f1ff000d000 nid=0x2b7 waiting on condition [0x00007f1ffb7fa000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c001c160> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "ignite-client-exec-3-2" #23 daemon prio=5 os_prio=0 tid=0x00007f20001ae000
> nid=0x2b6 waiting on condition [0x00007f1ffb8fb000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c001bdb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-client-3-#4%gridClient%" #22 prio=5 os_prio=0
> tid=0x00007f200017c800 nid=0x2b5 runnable [0x00007f1ffbbfc000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000c0007f00> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c0007ef0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c0007f20> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-client-2-#3%gridClient%" #21 prio=5 os_prio=0
> tid=0x00007f200017a800 nid=0x2b4 runnable [0x00007f1ffbcfd000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000c001c610> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c001c600> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c001c630> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-client-1-#2%gridClient%" #20 prio=5 os_prio=0
> tid=0x00007f2000179000 nid=0x2b3 runnable [0x00007f1ffbdfe000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000c00237f8> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c003a000> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c00236e0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "grid-nio-worker-tcp-client-0-#1%gridClient%" #19 prio=5 os_prio=0
> tid=0x00007f2000177000 nid=0x2b2 runnable [0x00007f201812a000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000c003aa90> (a
> org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
>         - locked <0x00000000c003d658> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000c003a9e8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
>         at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "pool-2-thread-3" #16 prio=5 os_prio=0 tid=0x00007f1ffc001000 nid=0x2b0
> waiting on condition [0x00007f2018331000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000000c0006268> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - None
>
> "pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007f2004007000 nid=0x2af in
> Object.wait() [0x00007f2018432000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
> org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
>         - locked <0x00000000c000fb00> (a java.util.LinkedList)
>         - locked <0x00000000c000fb00> (a java.util.LinkedList)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000c00c82c8> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007f2004005000 nid=0x2ae in
> Object.wait() [0x00007f2018533000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
> org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
>         - locked <0x00000000c000f958> (a java.util.LinkedList)
>         - locked <0x00000000c000f958> (a java.util.LinkedList)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000c00c85d8> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "pool-1-thread-3" #13 prio=5 os_prio=0 tid=0x00007f2014005000 nid=0x2ad
> runnable [0x00007f2018634000]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>         at java.net.SocketInputStream.read(SocketInputStream.java:171)
>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>         - locked <0x00000000c00c8cf8> (a java.io.BufferedInputStream)
>         at
>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>         at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>         at
>
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>         at
>
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>         at
>
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>         at
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000c00c8f30> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "pool-1-thread-2" #12 prio=5 os_prio=0 tid=0x00007f2014003800 nid=0x2ac in
> Object.wait() [0x00007f2018948000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:357)
>         - locked <0x00000000c00804d8> (a java.util.LinkedList)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:970)
>         at
>
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:2000)
>         at
>
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1985)
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000c00c9808> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "pool-1-thread-1" #11 prio=5 os_prio=0 tid=0x00007f2014002000 nid=0x2ab in
> Object.wait() [0x00007f2018a49000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:447)
>         - locked <0x00000000c00060b0> (a
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJobListener)
>         at
>
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1859)
>         at
>
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1844)
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>    Locked ownable synchronizers:
>         - <0x00000000c00c9080> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "Thread-0" #9 prio=5 os_prio=0 tid=0x00007f20645ef000 nid=0x2a6 runnable
> [0x00007f2018b4a000]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>         at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
>         at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>         at java.net.ServerSocket.accept(ServerSocket.java:513)
>         at
>
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>         at
>
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
>         at
>
> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:203)
>
>    Locked ownable synchronizers:
>         - None
>
> "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f20640c2000
> nid=0x2a4
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f20640bf000
> nid=0x2a3 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f20640bd000
> nid=0x2a2 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f20640ba000
> nid=0x2a1 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f20640b8800
> nid=0x2a0 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>         - None
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2064086800 nid=0x29f in
> Object.wait() [0x00007f2040ad0000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>         - locked <0x00000000c00ceaf8> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)
>
>    Locked ownable synchronizers:
>         - None
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2064082000
> nid=0x29e in Object.wait() [0x00007f2040bd1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>         - locked <0x00000000c00ced28> (a java.lang.ref.Reference$Lock)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
>    Locked ownable synchronizers:
>         - None
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f206400c800 nid=0x298 in
> Object.wait()
> [0x00007f206bab0000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000c007e0f0> (a
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
>         at java.lang.Thread.join(Thread.java:1252)
>         - locked <0x00000000c007e0f0> (a
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
>         at java.lang.Thread.join(Thread.java:1326)
>         at
>
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:274)
>
>    Locked ownable synchronizers:
>         - None
>
> "VM Thread" os_prio=0 tid=0x00007f206407a000 nid=0x29d runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f2064022000 nid=0x299
> runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f2064023800 nid=0x29a
> runnable
>
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f2064025800 nid=0x29b
> runnable
>
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f2064027000 nid=0x29c
> runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f20640c4800 nid=0x2a5
> waiting
> on condition
>
> JNI global references: 206
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[sv <sv...@gmail.com>]

Evgenii,

1) How do I enable DEBUG logging on the Ignite cluster?

2) I used jstack to produce a thread dump of the Zeppelin Ignite interpreter
(which runs in a separate process).  I pasted it below.  Does anything look
out of place to you?

2018-11-03 09:32:37
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #82 daemon prio=9 os_prio=0 tid=0x00007f202c001000
nid=0x37b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"tcp-client-disco-msg-worker-#4" #81 prio=5 os_prio=0 tid=0x00007f1fd1f2a000
nid=0x2fb waiting on condition [0x00007f1fce1e6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:522)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1561)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-reader-#3" #80 prio=5 os_prio=0
tid=0x00007f1fd1f25000 nid=0x2fa in Object.wait() [0x00007f1fce2e7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:1006)
        - locked <0x00000000ee2bc268> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"tcp-client-disco-sock-writer-#2" #79 prio=5 os_prio=0
tid=0x00007f1fd1f24800 nid=0x2f9 in Object.wait() [0x00007f1fce3e8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1209)
        - locked <0x00000000ee2bbd88> (a java.lang.Object)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"TcpDiscoverySpi.timer" #78 prio=5 os_prio=0 tid=0x00007f1fd1f20800
nid=0x2f8 in Object.wait() [0x00007f1fce4e9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000ee228468> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
        - None

"grid-data-loader-flusher-#39" #77 prio=5 os_prio=0 tid=0x00007f1fd1efb000
nid=0x2f7 waiting on condition [0x00007f1fce5ea000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000edc52cc0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-3-#37" #75 prio=5 os_prio=0
tid=0x00007f1fd1eea800 nid=0x2f6 runnable [0x00007f1fce8eb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edb0fdf0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed992ff0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed992ed8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-2-#36" #74 prio=5 os_prio=0
tid=0x00007f1fd1ee8800 nid=0x2f5 runnable [0x00007f1fce9ec000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed98e750> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed97e4f8> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed97e3e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-1-#35" #73 prio=5 os_prio=0
tid=0x00007f1fd1ee7000 nid=0x2f4 runnable [0x00007f1fceaed000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed979c58> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed969a00> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9698e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-client-listener-0-#34" #72 prio=5 os_prio=0
tid=0x00007f1fd1e67800 nid=0x2f3 runnable [0x00007f1fcebee000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed965160> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000ed954f08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed954df0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#33" #71 prio=5 os_prio=0 tid=0x00007f1fd1e66800 nid=0x2f2
runnable [0x00007f1fcecef000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ed944218> (a sun.nio.ch.Util$3)
        - locked <0x00000000ed944208> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ed9440f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"tcp-comm-worker-#1" #70 prio=5 os_prio=0 tid=0x00007f1fd1cdf800 nid=0x2f1
waiting on condition [0x00007f1fcf1f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000fd641060> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4207)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-3-#32" #66 prio=5 os_prio=0 tid=0x00007f1fd1cdd800
nid=0x2f0 runnable [0x00007f1fcf2f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda143b0> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda16430> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda14308> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-2-#31" #65 prio=5 os_prio=0 tid=0x00007f1fd1cd7000
nid=0x2ef runnable [0x00007f1fcf6f5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda16b38> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda49580> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda16a90> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-1-#30" #64 prio=5 os_prio=0 tid=0x00007f1fd1cd8800
nid=0x2ee runnable [0x00007f1fcf4f3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda49c88> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4bd08> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda49be0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-comm-0-#29" #63 prio=5 os_prio=0 tid=0x00007f1fd1cd5800
nid=0x2ed runnable [0x00007f1fcf5f4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda4c410> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000fda4e490> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda4c368> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"nio-acceptor-#28" #62 prio=5 os_prio=0 tid=0x00007f1fd1cdb000 nid=0x2ec
runnable [0x00007f1fcf3f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000fda13c60> (a sun.nio.ch.Util$3)
        - locked <0x00000000fda13c50> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000fda13838> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2870)
        at
org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2822)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-timeout-worker-#27" #61 prio=5 os_prio=0 tid=0x00007f1fd1c4b800
nid=0x2e8 in Object.wait() [0x00007f1fcf7f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:203)
        - locked <0x00000000fd9add58> (a java.lang.Object)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-clock" #60 daemon prio=10 os_prio=0 tid=0x00007f1fd1c3f000 nid=0x2e7
sleeping[0x00007f1fcf8f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.util.IgniteUtils$22.run(IgniteUtils.java:3308)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"jvm-pause-detector-worker" #53 daemon prio=5 os_prio=0
tid=0x00007f1fd184e800 nid=0x2e0 sleeping[0x00007f1fcfffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-7-#20" #52 prio=5 os_prio=0 tid=0x00007f1fd1805800
nid=0x2df waiting on condition [0x00007f1ff836b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-6-#19" #51 prio=5 os_prio=0 tid=0x00007f1fd1803800
nid=0x2de waiting on condition [0x00007f1ff846c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-5-#18" #50 prio=5 os_prio=0 tid=0x00007f1fd1802000
nid=0x2dd waiting on condition [0x00007f1ff856d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-4-#17" #49 prio=5 os_prio=0 tid=0x00007f1fd1800000
nid=0x2dc waiting on condition [0x00007f1ff866e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-3-#16" #48 prio=5 os_prio=0 tid=0x00007f1fd17fe000
nid=0x2db waiting on condition [0x00007f1ff876f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-2-#15" #47 prio=5 os_prio=0 tid=0x00007f1fd17fc000
nid=0x2da waiting on condition [0x00007f1ff8870000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-1-#14" #46 prio=5 os_prio=0 tid=0x00007f1fd17fa800
nid=0x2d9 waiting on condition [0x00007f1ff8971000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"data-streamer-stripe-0-#13" #45 prio=5 os_prio=0 tid=0x00007f1fd17f8800
nid=0x2d8 waiting on condition [0x00007f1ff8a72000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-7-#12" #44 prio=5 os_prio=0 tid=0x00007f1fd17f6800 nid=0x2d7
waiting on condition [0x00007f1ff8b73000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-6-#11" #43 prio=5 os_prio=0 tid=0x00007f1fd17f4800 nid=0x2d6
waiting on condition [0x00007f1ff8c74000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-5-#10" #42 prio=5 os_prio=0 tid=0x00007f1fd17f2800 nid=0x2d5
waiting on condition [0x00007f1ff8d75000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-4-#9" #41 prio=5 os_prio=0 tid=0x00007f1fd17f0800 nid=0x2d4
waiting on condition [0x00007f1ff8e76000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-3-#8" #40 prio=5 os_prio=0 tid=0x00007f1fd17ee800 nid=0x2d3
waiting on condition [0x00007f1ff8f77000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-2-#7" #39 prio=5 os_prio=0 tid=0x00007f1fd17d3000 nid=0x2d2
waiting on condition [0x00007f1ff9078000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-1-#6" #38 prio=5 os_prio=0 tid=0x00007f1fd17d1800 nid=0x2d1
waiting on condition [0x00007f1ff9179000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"sys-stripe-0-#5" #37 prio=5 os_prio=0 tid=0x00007f1fd17cf000 nid=0x2d0
waiting on condition [0x00007f1ffa0f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:672)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"JobProgressPoller, jobId=20181102-071721_1550658378" #36 prio=5 os_prio=0
tid=0x00007f1fd0003000 nid=0x2cf waiting on condition [0x00007f1ffadf2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:61)

   Locked ownable synchronizers:
        - None

"pool-2-thread-7" #35 prio=5 os_prio=0 tid=0x00007f2008001000 nid=0x2ce
waiting on condition [0x00007f1ffaef3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-6" #33 prio=5 os_prio=0 tid=0x00007f1ffc005800 nid=0x2c9
waiting on condition [0x00007f1ffb1f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-5" #31 prio=5 os_prio=0 tid=0x00007f1ffc004800 nid=0x2c4
waiting on condition [0x00007f1ffb2f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee2280f0> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:271)
        at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
        at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
        at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
        at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
        - locked <0x00000000fd9eaa10> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
        at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
        at org.apache.ignite.Ignition.start(Ignition.java:327)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.getIgnite(IgniteInterpreter.java:190)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.initIgnite(IgniteInterpreter.java:208)
        at
org.apache.zeppelin.ignite.IgniteInterpreter.open(IgniteInterpreter.java:127)
        at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
        - locked <0x00000000c000f738> (a
org.apache.zeppelin.ignite.IgniteInterpreter)
        - locked <0x00000000c000f810> (a
org.apache.zeppelin.interpreter.LazyOpenInterpreter)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
        at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000fda11fc8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12080> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda12138> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000fda121f0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        - <0x00000000c000f900> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-4" #29 prio=5 os_prio=0 tid=0x00007f1ffc003000 nid=0x2bf
waiting on condition [0x00007f1ffb3f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-5" #28 daemon prio=5 os_prio=0 tid=0x00007f1fd0001000
nid=0x2bb waiting on condition [0x00007f1ffb4f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-4" #27 daemon prio=5 os_prio=0 tid=0x00007f1fe4001000
nid=0x2ba waiting on condition [0x00007f2018230000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-3" #26 daemon prio=5 os_prio=0 tid=0x00007f20001c9000
nid=0x2b9 waiting on condition [0x00007f1ffb5f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"a65ff0b2-c2ba-4d1c-8d67-25570203a476-topology-update" #25 daemon prio=5
os_prio=0 tid=0x00007f20001c5800 nid=0x2b8 waiting on condition
[0x00007f1ffb6f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.ignite.internal.client.impl.GridClientImpl$TopologyUpdaterThread.run(GridClientImpl.java:521)

   Locked ownable synchronizers:
        - None

"ignite-client-top-lsnr-1-2" #24 daemon prio=5 os_prio=0
tid=0x00007f1ff000d000 nid=0x2b7 waiting on condition [0x00007f1ffb7fa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001c160> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"ignite-client-exec-3-2" #23 daemon prio=5 os_prio=0 tid=0x00007f20001ae000
nid=0x2b6 waiting on condition [0x00007f1ffb8fb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c001bdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-3-#4%gridClient%" #22 prio=5 os_prio=0
tid=0x00007f200017c800 nid=0x2b5 runnable [0x00007f1ffbbfc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c0007f00> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c0007ef0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c0007f20> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-2-#3%gridClient%" #21 prio=5 os_prio=0
tid=0x00007f200017a800 nid=0x2b4 runnable [0x00007f1ffbcfd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c001c610> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c001c600> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c001c630> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-1-#2%gridClient%" #20 prio=5 os_prio=0
tid=0x00007f2000179000 nid=0x2b3 runnable [0x00007f1ffbdfe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c00237f8> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003a000> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c00236e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"grid-nio-worker-tcp-client-0-#1%gridClient%" #19 prio=5 os_prio=0
tid=0x00007f2000177000 nid=0x2b2 runnable [0x00007f201812a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000c003aa90> (a
org.apache.ignite.internal.util.nio.SelectedSelectionKeySet)
        - locked <0x00000000c003d658> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000c003a9e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2105)
        at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-3" #16 prio=5 os_prio=0 tid=0x00007f1ffc001000 nid=0x2b0
waiting on condition [0x00007f2018331000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c0006268> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007f2004007000 nid=0x2af in
Object.wait() [0x00007f2018432000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        - locked <0x00000000c000fb00> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c82c8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007f2004005000 nid=0x2ae in
Object.wait() [0x00007f2018533000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:113)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        - locked <0x00000000c000f958> (a java.util.LinkedList)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c85d8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-3" #13 prio=5 os_prio=0 tid=0x00007f2014005000 nid=0x2ad
runnable [0x00007f2018634000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000000c00c8cf8> (a java.io.BufferedInputStream)
        at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c8f30> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-2" #12 prio=5 os_prio=0 tid=0x00007f2014003800 nid=0x2ac in
Object.wait() [0x00007f2018948000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:357)
        - locked <0x00000000c00804d8> (a java.util.LinkedList)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:970)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:2000)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1985)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9808> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"pool-1-thread-1" #11 prio=5 os_prio=0 tid=0x00007f2014002000 nid=0x2ab in
Object.wait() [0x00007f2018a49000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:447)
        - locked <0x00000000c00060b0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJobListener)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1859)
        at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1844)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000c00c9080> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

"Thread-0" #9 prio=5 os_prio=0 tid=0x00007f20645ef000 nid=0x2a6 runnable
[0x00007f2018b4a000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
        at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:203)

   Locked ownable synchronizers:
        - None

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f20640c2000 nid=0x2a4
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f20640bf000
nid=0x2a3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f20640bd000
nid=0x2a2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f20640ba000
nid=0x2a1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f20640b8800
nid=0x2a0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f2064086800 nid=0x29f in
Object.wait() [0x00007f2040ad0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000c00ceaf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f2064082000
nid=0x29e in Object.wait() [0x00007f2040bd1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000000c00ced28> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x00007f206400c800 nid=0x298 in Object.wait()
[0x00007f206bab0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1252)
        - locked <0x00000000c007e0f0> (a
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1326)
        at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:274)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=0 tid=0x00007f206407a000 nid=0x29d runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f2064022000 nid=0x299
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f2064023800 nid=0x29a
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f2064025800 nid=0x29b
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f2064027000 nid=0x29c
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f20640c4800 nid=0x2a5 waiting
on condition

JNI global references: 206




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[Evgenii Zhuravlev <e....@gmail.com>]

IGNITE_QUIET=false enables only INFO logging. Well, then something stuck on
the Zeppelin side, still it worth to collect the thread dumps from it.

Evgenii

пт, 2 нояб. 2018 г. в 10:39, sv <sv...@gmail.com>:

> Actually, I already am running each Ignite node with IGNITE_QUIET=false, so
> that should already have INFO and DEBUG logging enabled.  But still,
> nothing.  I suspect that the Zeppelin Ignite interpreter is not reaching
> the
> Ignite cluster.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[sv <sv...@gmail.com>]

Actually, I already am running each Ignite node with IGNITE_QUIET=false, so
that should already have INFO and DEBUG logging enabled.  But still,
nothing.  I suspect that the Zeppelin Ignite interpreter is not reaching the
Ignite cluster.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[sv <sv...@gmail.com>]

Evgenii,
I am not sure if it even gets to the Ignite cluster.

Here is what I see in Zeppelin interpreter log, when I try to execute scala
code in a notebook:  

 INFO [2018-11-02 07:29:38,005] ({pool-2-thread-4}
SchedulerFactory.java[jobStarted]:109) - Job 20181102-071721_1550658378
started by scheduler org.apache.zeppelin.ignite.IgniteInterpreter1313679265

After that, nothing.

There is absolutely nothing in Ignite logs.

I have not increased the log level for Ignite, as that would mean I would
have to reload the entire cluster.  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ouch! Argument is invalid: Cache name must not be null or empty.

[Evgenii Zhuravlev <e....@gmail.com>]

> executing scala code (using %ignite) makes the entire Ignite interpreter
in Zeppelin hang, without any errors in logs.  How do I debug this?
When something hangs, you have to check thread dumps to understand where
and why it hangs. Also, sometimes, it worth to enable another level of the
logging - INFO or even DEBUG.

Evgenii

чт, 1 нояб. 2018 г. в 11:05, sv <sv...@gmail.com>:

> Hi, to clarify:
> 1) Thin client works fine
> 2) Thick client using port 11211 works fine
> 3) The "Client Node" connection is the one that was not working.
>
> I got the Client Node connection (#3) to work by specifying the cache name
> in the connection string.  Now it connects, and all SQL works well.
>
> I thought that this was fixed already in 2.6, so that you didn't have to
> specify the cache name anymore?  In any case, it works.
>
> The next problem that I ran into:  executing scala code (using %ignite)
> makes the entire Ignite interpreter in Zeppelin hang, without any errors in
> logs.  How do I debug this?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>