[jira] [Commented] (LIVY-487) Livy remote jars and files incorrectly specified on classpath

["Ruslan Dautkhanov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760032#comment-16760032 ] 

Ruslan Dautkhanov commented on LIVY-487:
----------------------------------------

If you specify dependencies to spark-submit as `--jars`, they get distributed to executors (and driver in yarn-cluster mode). They are stored in a temp directory specific for that yarn job.. 

I thought Livy uses the same approach.. 

How did you add dependencies to your job? 

> Livy remote jars and files incorrectly specified on classpath
> -------------------------------------------------------------
>
>                 Key: LIVY-487
>                 URL: https://issues.apache.org/jira/browse/LIVY-487
>             Project: Livy
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.5.0
>            Reporter: Tim Harsch
>            Priority: Critical
>
> From a thread in users-livy group:
> If I submit jars using a client machine that is not the same as the server (so file system is not shared) and I submit the jar it’s looks like it gets transmitted to the server but then I can’t reference any of the classes.  *If I look at the Spark UI that gets launched for the session then I can see references to paths in my client machine*, which aren’t resolvable on the server.   Is this the scenario you have?   I think it might occur in a docker env if you are doing client stuff on the host os, and livy is running in a container.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)