Read a HDFS file from Spark source code

[rapelly kartheek <ka...@gmail.com>]

Hi

I am trying to access a file in HDFS from spark "source code". Basically, I
am tweaking the spark source code. I need to access a file in HDFS from the
source code of the spark. I am really not understanding how to go about
doing this.

Can someone please help me out in this regard.
Thank you!!
Karthik

Re: Read a HDFS file from Spark source code

[rapelly kartheek <ka...@gmail.com>]

Hi Sean,
I was following this link;

http://mund-consulting.com/Blog/Posts/file-operations-in-HDFS-using-java.aspx

But, I was facing FileSystem ambiguity error. I really don't have any idea
as to how to go about doing this.
Can you please help me how to start off with this?


On Wed, Nov 12, 2014 at 11:26 AM, Samarth Mailinglist <
mailinglistsamarth@gmail.com> wrote:

> Instead of a file path, use a HDFS URI.
> For example: (In Python)
>
>
>
> data = sc.textFile("hdfs://localhost/user/someuser/data")
>
> ​
>
> On Wed, Nov 12, 2014 at 10:12 AM, rapelly kartheek <
> kartheek.mbms@gmail.com> wrote:
>
>> Hi
>>
>> I am trying to access a file in HDFS from spark "source code". Basically,
>> I am tweaking the spark source code. I need to access a file in HDFS from
>> the source code of the spark. I am really not understanding how to go about
>> doing this.
>>
>> Can someone please help me out in this regard.
>> Thank you!!
>> Karthik
>>
>
>

Re: Read a HDFS file from Spark source code

[Samarth Mailinglist <ma...@gmail.com>]

Instead of a file path, use a HDFS URI.
For example: (In Python)



data = sc.textFile("hdfs://localhost/user/someuser/data")

​

On Wed, Nov 12, 2014 at 10:12 AM, rapelly kartheek <ka...@gmail.com>
wrote:

> Hi
>
> I am trying to access a file in HDFS from spark "source code". Basically,
> I am tweaking the spark source code. I need to access a file in HDFS from
> the source code of the spark. I am really not understanding how to go about
> doing this.
>
> Can someone please help me out in this regard.
> Thank you!!
> Karthik
>