[jira] [Created] (ZEPPELIN-5127) Livy Interpreter Failing to Connect to Remote EMR

["Andy (Jira)" <ji...@apache.org>]

Andy created ZEPPELIN-5127:
------------------------------

             Summary: Livy Interpreter Failing to Connect to Remote EMR
                 Key: ZEPPELIN-5127
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5127
             Project: Zeppelin
          Issue Type: Bug
            Reporter: Andy
         Attachments: Screen Shot 2020-11-09 at 5.55.20 PM.png

Hi,

We have an EMR cluster in AWS running the Livy server. I installed the Zeppelin binary to my local, and added the master node's IP as the URL to the interpreter settings. Screenshot is attached.

Livy server settings ({{livy.conf}}) on the cluster are set to defaults.


The issue is, whenever I try to run a {{%livy.spark}} cell, it always times out when trying to check the Livy server version on the remote cluster. This is the error message outputted:
{noformat}
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to create session, please check livy interpreter log and livy server log
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
	at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
	at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to create session, please check livy interpreter log and livy server log
	at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:169)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
	... 8 more
Caused by: org.apache.zeppelin.livy.LivyException: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://REDACTED:8998/version": Operation timed out; nested exception is java.net.ConnectException: Operation timed out
	at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:743)
	at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:696)
	at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivyVersion(BaseLivyInterpreter.java:450)
	at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:159)
	... 9 more
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://REDACTED:8998/version": Operation timed out; nested exception is java.net.ConnectException: Operation timed out
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:633)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
	at org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:716)
	... 12 more
Caused by: java.net.ConnectException: Operation timed out
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:574)
	at java.base/sun.nio.ch.Net.connect(Net.java:563)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
	at java.base/java.net.Socket.connect(Socket.java:648)
	at java.base/java.net.Socket.connect(Socket.java:597)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1261)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1016)
	at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:619)
	... 15 more{noformat}
 

One thing to note is that I'm behind a corporate proxy, but I set the env vars before starting Zeppelin, and when I try curling the URL in a {{%sh}} cell, it works fine.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)