You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Divya Arya <di...@cyberops.in> on 2018/07/26 12:19:47 UTC
Query: Local Path cannot read by Livy (Running with Docker)
Hello Team,
I need a little support from you, I am trying to bundle up my application
which is working on Apache Livy, Spark (SCALA), so it can run on a
Windows-based system, for that, I am using Apache Docker.
I have set up Livy and Spark in that, which is working perfectly fine but I
am not able to use my Local path files (suppose files in C:/ or D:/) in
SparkSession and SparkContext. Please help me out here.
Thanks and Regards,
Divya Arya
divya@cyberops.in
divya.arya69@gmail.com
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by ka...@gmail.com.
Docker is not windows! It is a cgroup in Linux. So no C: or D:
Sent from my iPhone
> On Jul 26, 2018, at 11:44 PM, Divya Arya <di...@cyberops.in> wrote:
>
> Please help me out. I am really looking forward.
>
>> On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in> wrote:
>> Hello Team,
>>
>> I need a little support from you, I am trying to bundle up my application which is working on Apache Livy, Spark (SCALA), so it can run on a Windows-based system, for that, I am using Apache Docker.
>>
>> I have set up Livy and Spark in that, which is working perfectly fine but I am not able to use my Local path files (suppose files in C:/ or D:/) in SparkSession and SparkContext. Please help me out here.
>>
>>
>> Thanks and Regards,
>>
>> Divya Arya
>>
>> divya@cyberops.in
>> divya.arya69@gmail.com
>>
>
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by Divya Arya <di...@cyberops.in>.
I got a patch for that but it is *not a valid solution*. I installed
xampp on my windows system and shared my C drive, (few changes in the
configuration file) as a local server then access those paths.
On Mon, Jul 30, 2018 at 9:56 PM, Harsch, Tim <Ti...@teradata.com>
wrote:
> I created a Jira to start tracking the issue:
>
> https://issues.apache.org/jira/browse/LIVY-487
>
>
> When you see the issue are you using the files parameter of sessions
> endpoint?
>
>
> It would be helpful for the ticket if we could construct SSCCE (
> http://sscce.org/ ) to demonstrate the problem. I don't have an
> environment going where I can do that easily at the moment. If you do, can
> you please comment on the ticket?
>
>
> Thanks,
>
> Tim
>
> ------------------------------
> *From:* Divya Arya <di...@cyberops.in>
> *Sent:* Friday, July 27, 2018 11:17:30 PM
> *To:* user@livy.incubator.apache.org
> *Subject:* Re: Query: Local Path cannot read by Livy (Running with Docker)
>
> [External Email]
>
>
> Yes, that is the exact same scenario I have, just instead of jar files I
> am using CSV files stored in my client machine's file system and no
> protocol is supporting to access those files.
>
> On Sat, Jul 28, 2018 at 3:42 AM, Harsch, Tim <Ti...@teradata.com>
> wrote:
>
> When you say not able to use my files in Spark Session what do you mean?
>
>
>
> I think I’m seeing a similar problem. If I submit jars using a client
> machine that is not the same as the server (so file system is not shared)
> and I submit the jar it’s looks like it gets transmitted to the server but
> then I can’t reference any of the classes. If I look at the Spark UI that
> gets launched for the session then I can see references to paths in my
> client machine, which aren’t resolvable on the server. Is this the
> scenario you have? I think it might occur in a docker env if you are
> doing client stuff on the host os, and livy is running in a container.
>
>
>
>
>
>
>
> *From: *Divya Arya <di...@cyberops.in>
> *Reply-To: *"user@livy.incubator.apache.org" <
> user@livy.incubator.apache.org>
> *Date: *Thursday, July 26, 2018 at 11:45 PM
> *To: *"user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
> *Subject: *Re: Query: Local Path cannot read by Livy (Running with Docker)
>
>
>
> *[External Email] *
>
>
>
> Please help me out. I am really looking forward.
>
>
>
> On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in> wrote:
>
> Hello Team,
>
>
>
> I need a little support from you, I am trying to bundle up my application
> which is working on Apache Livy, Spark (SCALA), so it can run on a
> Windows-based system, for that, I am using Apache Docker.
>
>
>
> I have set up Livy and Spark in that, which is working perfectly fine but
> I am not able to use my Local path files (suppose files in C:/ or D:/) in
> SparkSession and SparkContext. Please help me out here.
>
>
>
>
>
> Thanks and Regards,
>
>
>
> Divya Arya
>
>
>
> divya@cyberops.in
>
> divya.arya69@gmail.com
>
>
>
>
>
>
>
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by "Harsch, Tim" <Ti...@Teradata.com>.
I created a Jira to start tracking the issue:
https://issues.apache.org/jira/browse/LIVY-487
When you see the issue are you using the files parameter of sessions endpoint?
It would be helpful for the ticket if we could construct SSCCE ( http://sscce.org/ ) to demonstrate the problem. I don't have an environment going where I can do that easily at the moment. If you do, can you please comment on the ticket?
Thanks,
Tim
________________________________
From: Divya Arya <di...@cyberops.in>
Sent: Friday, July 27, 2018 11:17:30 PM
To: user@livy.incubator.apache.org
Subject: Re: Query: Local Path cannot read by Livy (Running with Docker)
[External Email]
Yes, that is the exact same scenario I have, just instead of jar files I am using CSV files stored in my client machine's file system and no protocol is supporting to access those files.
On Sat, Jul 28, 2018 at 3:42 AM, Harsch, Tim <Ti...@teradata.com>> wrote:
When you say not able to use my files in Spark Session what do you mean?
I think I’m seeing a similar problem. If I submit jars using a client machine that is not the same as the server (so file system is not shared) and I submit the jar it’s looks like it gets transmitted to the server but then I can’t reference any of the classes. If I look at the Spark UI that gets launched for the session then I can see references to paths in my client machine, which aren’t resolvable on the server. Is this the scenario you have? I think it might occur in a docker env if you are doing client stuff on the host os, and livy is running in a container.
From: Divya Arya <di...@cyberops.in>>
Reply-To: "user@livy.incubator.apache.org<ma...@livy.incubator.apache.org>" <us...@livy.incubator.apache.org>>
Date: Thursday, July 26, 2018 at 11:45 PM
To: "user@livy.incubator.apache.org<ma...@livy.incubator.apache.org>" <us...@livy.incubator.apache.org>>
Subject: Re: Query: Local Path cannot read by Livy (Running with Docker)
[External Email]
Please help me out. I am really looking forward.
On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in>> wrote:
Hello Team,
I need a little support from you, I am trying to bundle up my application which is working on Apache Livy, Spark (SCALA), so it can run on a Windows-based system, for that, I am using Apache Docker.
I have set up Livy and Spark in that, which is working perfectly fine but I am not able to use my Local path files (suppose files in C:/ or D:/) in SparkSession and SparkContext. Please help me out here.
Thanks and Regards,
Divya Arya
divya@cyberops.in<ma...@cyberops.in>
divya.arya69@gmail.com<ma...@gmail.com>
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by Divya Arya <di...@cyberops.in>.
Yes, that is the exact same scenario I have, just instead of jar files I am
using CSV files stored in my client machine's file system and no protocol
is supporting to access those files.
On Sat, Jul 28, 2018 at 3:42 AM, Harsch, Tim <Ti...@teradata.com>
wrote:
> When you say not able to use my files in Spark Session what do you mean?
>
>
>
> I think I’m seeing a similar problem. If I submit jars using a client
> machine that is not the same as the server (so file system is not shared)
> and I submit the jar it’s looks like it gets transmitted to the server but
> then I can’t reference any of the classes. If I look at the Spark UI that
> gets launched for the session then I can see references to paths in my
> client machine, which aren’t resolvable on the server. Is this the
> scenario you have? I think it might occur in a docker env if you are
> doing client stuff on the host os, and livy is running in a container.
>
>
>
>
>
>
>
> *From: *Divya Arya <di...@cyberops.in>
> *Reply-To: *"user@livy.incubator.apache.org" <user@livy.incubator.apache.
> org>
> *Date: *Thursday, July 26, 2018 at 11:45 PM
> *To: *"user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
> *Subject: *Re: Query: Local Path cannot read by Livy (Running with Docker)
>
>
>
> *[External Email] *
>
>
>
> Please help me out. I am really looking forward.
>
>
>
> On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in> wrote:
>
> Hello Team,
>
>
>
> I need a little support from you, I am trying to bundle up my application
> which is working on Apache Livy, Spark (SCALA), so it can run on a
> Windows-based system, for that, I am using Apache Docker.
>
>
>
> I have set up Livy and Spark in that, which is working perfectly fine but
> I am not able to use my Local path files (suppose files in C:/ or D:/) in
> SparkSession and SparkContext. Please help me out here.
>
>
>
>
>
> Thanks and Regards,
>
>
>
> Divya Arya
>
>
>
> divya@cyberops.in
>
> divya.arya69@gmail.com
>
>
>
>
>
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by "Harsch, Tim" <Ti...@Teradata.com>.
When you say not able to use my files in Spark Session what do you mean?
I think I’m seeing a similar problem. If I submit jars using a client machine that is not the same as the server (so file system is not shared) and I submit the jar it’s looks like it gets transmitted to the server but then I can’t reference any of the classes. If I look at the Spark UI that gets launched for the session then I can see references to paths in my client machine, which aren’t resolvable on the server. Is this the scenario you have? I think it might occur in a docker env if you are doing client stuff on the host os, and livy is running in a container.
From: Divya Arya <di...@cyberops.in>
Reply-To: "user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
Date: Thursday, July 26, 2018 at 11:45 PM
To: "user@livy.incubator.apache.org" <us...@livy.incubator.apache.org>
Subject: Re: Query: Local Path cannot read by Livy (Running with Docker)
[External Email]
Please help me out. I am really looking forward.
On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in>> wrote:
Hello Team,
I need a little support from you, I am trying to bundle up my application which is working on Apache Livy, Spark (SCALA), so it can run on a Windows-based system, for that, I am using Apache Docker.
I have set up Livy and Spark in that, which is working perfectly fine but I am not able to use my Local path files (suppose files in C:/ or D:/) in SparkSession and SparkContext. Please help me out here.
Thanks and Regards,
Divya Arya
divya@cyberops.in<ma...@cyberops.in>
divya.arya69@gmail.com<ma...@gmail.com>
Re: Query: Local Path cannot read by Livy (Running with Docker)
Posted by Divya Arya <di...@cyberops.in>.
Please help me out. I am really looking forward.
On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <di...@cyberops.in> wrote:
> Hello Team,
>
> I need a little support from you, I am trying to bundle up my application
> which is working on Apache Livy, Spark (SCALA), so it can run on a
> Windows-based system, for that, I am using Apache Docker.
>
> I have set up Livy and Spark in that, which is working perfectly fine but
> I am not able to use my Local path files (suppose files in C:/ or D:/) in
> SparkSession and SparkContext. Please help me out here.
>
>
> Thanks and Regards,
>
> Divya Arya
>
> divya@cyberops.in
> divya.arya69@gmail.com
>
>