Spark job via livy issue from a docker container

[sarjeet singh <ss...@gmail.com>]

Hello,

I have built a livy docker container and trying to connect it to a YARN
cluster running outside of docker container host. When I am submitting a
spark job via livy inside the container (yarn-cluster mode) from Zeppelin
UI, I see the job is submitted (verified from RM UI and yarn container
logs) but job eventually fail with following errors (shown in zeppelin UI):

%livy.spark
sc.version

org.apache.zeppelin.livy.LivyException: Session 3 is finished, appId:
application_1508274923092_0022, log: [ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624),
at java.lang.Thread.run(Thread.java:748), , Shell output: main : command
provided 1, main : user is user1, main : requested yarn user is user1, , ,
Container exited with a non-zero exit code 15, Failing this attempt.
Failing the application.]
at
org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:230)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:119)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:101)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

From the yarn application logs, I found the following related to the job
submitted:

2017-10-19 12:44:59,423 INFO [main] yarn.ApplicationMaster: Waiting for
spark context initialization... 2017-10-19 12:44:59,476 INFO [Driver]
driver.RSCDriver: Connecting to: 57539c730d5f:34680 2017-10-19 12:44:59,476
INFO [Driver] driver.RSCDriver: Starting RPC server... 2017-10-19
12:44:59,688 WARN [Driver] rsc.RSCConf: Your hostname, node1.lab (*valid
hostname*), resolves to a loopback address, but we couldn't find any
external IP address! 2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Set
livy.rsc.rpc.server.address if you need to bind to another address.
2017-10-19 12:44:59,704 ERROR [Driver] yarn.ApplicationMaster: User class
threw exception: java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException
java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException at
io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41) at
com.cloudera.livy.rsc.driver.RSCDriver.initializeServer(RSCDriver.java:197)
at com.cloudera.livy.rsc.driver.RSCDriver.run(RSCDriver.java:326) at
com.cloudera.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
Caused by: java.nio.channels.UnresolvedAddressException at
sun.nio.ch.Net.checkAddress(Net.java:101) at
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) at
io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:242)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:540)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:525)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:507)
at
io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215) at
io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)

As from above, it seems there is issue with connecting back to container or
resolving address, but, the dns resolution seems working from the cluster
node and inside docker container. Any pointers on what else I can check and
what else could be the issue?

Note: if the spark job is submitted via spark-submit from the same docker
container in Yarn-Cluster mode, it completes without any issue.

Hence, the issue is only isolated to livy or any configuration I may be
missing here? Let me know if there is any other information helpful in
debugging the issue.

- Sarjeet Singh

Re: Spark job via livy issue from a docker container

[sarjeet singh <ss...@gmail.com>]

Meisam,

No, it's not in livy.conf (
https://github.com/cloudera/livy/blob/branch-0.3/conf/livy.conf). So, it
should be using default value.

To add more, I observed the above issue when docker container is using
"bridge" networking mode. "host" networking mode works fine for me in the
above scenario.

What should be used for livy.rsc.rpc.server.address in a docker container
for livy? Any pointer is much appreciated.

- Sarjeet Singh

On Fri, Oct 20, 2017 at 8:38 AM, Meisam Fathi <me...@gmail.com>
wrote:

> Hi Sarjeet!
>
> Is livy.rsc.rpc.server.address set in the conf/livy-client.conf? Or in
> conf/livy.conf if you are using an older version of Livy?
>
> Thanks,
> Meisam
>
> 2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Your hostname,
>> node1.lab (*valid hostname*), resolves to a loopback address, but we
>> couldn't find any external IP address!
>> 2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Set
>> livy.rsc.rpc.server.address if you need to bind to another address.
>> 2017-10-19 12:44:59,704 ERROR [Driver] yarn.ApplicationMaster:
>>
>
>
>
>> Hence, the issue is only isolated to livy or any configuration I may be
>> missing here? Let me know if there is any other information helpful in
>> debugging the issue.
>>
>> - Sarjeet Singh
>>
>

Re: Spark job via livy issue from a docker container

[Meisam Fathi <me...@gmail.com>]

Hi Sarjeet!

Is livy.rsc.rpc.server.address set in the conf/livy-client.conf? Or in
conf/livy.conf if you are using an older version of Livy?

Thanks,
Meisam

2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Your hostname, node1.lab
> (*valid hostname*), resolves to a loopback address, but we couldn't find
> any external IP address!
> 2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Set
> livy.rsc.rpc.server.address if you need to bind to another address.
> 2017-10-19 12:44:59,704 ERROR [Driver] yarn.ApplicationMaster:
>



> Hence, the issue is only isolated to livy or any configuration I may be
> missing here? Let me know if there is any other information helpful in
> debugging the issue.
>
> - Sarjeet Singh
>