Reading large files

[Vijayasarathy Kannan <kv...@vt.edu>]

​Hi,

Is there a way to read a large file, in parallel​/distributed way? I have a
single large binary file which I currently read on the driver program and
then distribute it to executors (using groupBy(), etc.). I want to know if
there's a way to make the executors each read a specific/unique portion of
the file or create RDDs of multiple portions of the file and finally union
them.

Thanks.

RE: Reading large files

["Ulanov, Alexander" <al...@hp.com>]

With binaryRecords it loads the file by line into RDD. With binaryFiles it provides an input stream, so it is up to you if you want to load everything into memory. Though documentation does not suggest to use this function for large files.

From: Vijayasarathy Kannan [mailto:kvijay@vt.edu]
Sent: Wednesday, May 06, 2015 2:23 PM
To: Ulanov, Alexander
Cc: user@spark.apache.org
Subject: Re: Reading large files

Thanks.

In both cases, does the driver need to have enough memory to contain the entire file? How do both these functions work when, for example, the binary file is 4G and available driver memory is lesser?

On Wed, May 6, 2015 at 1:54 PM, Ulanov, Alexander <al...@hp.com>> wrote:
SparkContext has two methods for reading binary files: binaryFiles (reads multiple binary files into RDD) and binaryRecords (reads separate lines of a single binary file into RDD). For example, I have a big binary file split into logical parts, so I can use “binaryFiles”. The possible problem is that the order of records between parts is not preserved, so I have to do sortBy afterwards.

Alexander

From: Vijayasarathy Kannan [mailto:kvijay@vt.edu<ma...@vt.edu>]
Sent: Wednesday, May 06, 2015 10:38 AM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Reading large files

​Hi,

Is there a way to read a large file, in parallel​/distributed way? I have a single large binary file which I currently read on the driver program and then distribute it to executors (using groupBy(), etc.). I want to know if there's a way to make the executors each read a specific/unique portion of the file or create RDDs of multiple portions of the file and finally union them.

Thanks.

Re: Reading large files

[Vijayasarathy Kannan <kv...@vt.edu>]

Thanks.

In both cases, does the driver need to have enough memory to contain the
entire file? How do both these functions work when, for example, the binary
file is 4G and available driver memory is lesser?

On Wed, May 6, 2015 at 1:54 PM, Ulanov, Alexander <al...@hp.com>
wrote:

>  SparkContext has two methods for reading binary files: binaryFiles
> (reads multiple binary files into RDD) and binaryRecords (reads separate
> lines of a single binary file into RDD). For example, I have a big binary
> file split into logical parts, so I can use “binaryFiles”. The possible
> problem is that the order of records between parts is not preserved, so I
> have to do sortBy afterwards.
>
>
>
> Alexander
>
>
>
> *From:* Vijayasarathy Kannan [mailto:kvijay@vt.edu]
> *Sent:* Wednesday, May 06, 2015 10:38 AM
> *To:* user@spark.apache.org
> *Subject:* Reading large files
>
>
>
> ​Hi,
>
>
>
> Is there a way to read a large file, in parallel​/distributed way? I have
> a single large binary file which I currently read on the driver program and
> then distribute it to executors (using groupBy(), etc.). I want to know if
> there's a way to make the executors each read a specific/unique portion of
> the file or create RDDs of multiple portions of the file and finally union
> them.
>
>
>
> Thanks.
>

RE: Reading large files

["Ulanov, Alexander" <al...@hp.com>]

SparkContext has two methods for reading binary files: binaryFiles (reads multiple binary files into RDD) and binaryRecords (reads separate lines of a single binary file into RDD). For example, I have a big binary file split into logical parts, so I can use “binaryFiles”. The possible problem is that the order of records between parts is not preserved, so I have to do sortBy afterwards.

Alexander

From: Vijayasarathy Kannan [mailto:kvijay@vt.edu]
Sent: Wednesday, May 06, 2015 10:38 AM
To: user@spark.apache.org
Subject: Reading large files

​Hi,

Is there a way to read a large file, in parallel​/distributed way? I have a single large binary file which I currently read on the driver program and then distribute it to executors (using groupBy(), etc.). I want to know if there's a way to make the executors each read a specific/unique portion of the file or create RDDs of multiple portions of the file and finally union them.

Thanks.