You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Lewis John McGibbney (JIRA)" <ji...@apache.org> on 2018/02/07 18:40:00 UTC
[jira] [Comment Edited] (LIVY-372) Livy Doesn't work with Spark on
Mesos in cluster mode
[ https://issues.apache.org/jira/browse/LIVY-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257492#comment-16257492 ]
Lewis John McGibbney edited comment on LIVY-372 at 2/7/18 6:39 PM:
-------------------------------------------------------------------
We are currently investigating use of Mesos with Livy, can you please detail if there is currently any support? It appears that Mesos cluster manager is not supported currently for Livy, from the mailing list it looks like folks have only tested on local and yarn mode. [~jjacob] FYI
was (Author: lewismc):
We are currently investigating use of Mesos with Livy, can you please detail if there is currently any support? It appears that Mesos cluster manager is not supported currently for Livy, from the mailing list it looks like folks have only tested on local and yarn mode.
> Livy Doesn't work with Spark on Mesos in cluster mode
> -----------------------------------------------------
>
> Key: LIVY-372
> URL: https://issues.apache.org/jira/browse/LIVY-372
> Project: Livy
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.3
> Environment: Spark-2.1.0, Mesos-1.2.0
> Reporter: Yehor Krivokon
> Priority: Major
>
> Client mode - it works.
> But it doesnt work with cluster mode:
> Added to livy.conf:
> livy.spark.master=mesos://node1:5050
> livy.spark.deployMode=cluster
> Added to hue.ini:
> [spark]
> livy_server_host=node1
> livy_server_port=8998
> Livy logs:
> 17/06/16 21:00:26 INFO StateStore$: Using BlackholeStateStore for recovery.
> 17/06/16 21:00:26 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/06/16 21:00:26 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/06/16 21:00:26 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/06/16 21:00:26 INFO WebServer: Starting server on http://node1:8998
> 17/06/16 22:04:04 INFO LivyServer: Shutting down Livy server.
> 17/06/16 22:04:07 INFO StateStore$: Using BlackholeStateStore for recovery.
> 17/06/16 22:04:07 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/06/16 22:04:07 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/06/16 22:04:07 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/06/16 22:04:07 INFO WebServer: Starting server on http://node1:8998
> 17/06/16 22:05:07 INFO InteractiveSession$: Creating LivyClient for sessionId: 0
> 17/06/16 22:05:07 WARN RSCConf: Your hostname, node1, resolves to a loopback address, but we couldn't find any external IP addre$
> 17/06/16 22:05:07 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. (edited)
> [5:01]
> 7/06/16 22:05:10 WARN ContextLauncher: Child process exited with code 1.
> 17/06/16 22:05:10 ERROR RSCClient: Failed to connect to context.
> java.io.IOException: Child process exited with code 1.
> at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:416)
> at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:490)
> at java.lang.Thread.run(Thread.java:748)
> 17/06/16 22:05:10 INFO RSCClient: Failing pending job 8a53d717-773c-434e-807f-3ceb46138a66 due to shutdown.
> 17/06/16 22:05:10 INFO InteractiveSession: Stopping InteractiveSession 0...
> 17/06/16 22:05:10 INFO InteractiveSession: Stopped InteractiveSession 0.
> 17/06/16 22:05:10 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.io.IOException: Ch$
> 17/06/16 22:05:12 INFO InteractiveSession: Stopping InteractiveSession 0...
> 17/06/16 22:05:12 INFO InteractiveSession: Stopped InteractiveSession 0.
> 17/06/16 22:12:45 WARN HttpParser: Illegal character 0x16 in state=START for buffer HeapByteBuffer@5bf799a9[p=1,l=171,c=16384,r=$
> 17/06/16 22:12:45 WARN HttpParser: badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@10e07e9e{r=0,c=false,a=IDLE,ur$
> 17/06/16 22:58:15 WARN HttpParser: Illegal character 0x16 in state=START for buffer HeapByteBuffer@5bf799a9[p=1,l=171,c=16384,r=$
> 17/06/16 22:58:15 WARN HttpParser: badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@2ef0a110{r=0,c=false,a=ID
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)