Load multiple parquet file as single RDD

[Mohnish Kodnani <mo...@gmail.com>]

Hi,
I am trying to play around with Spark and Spark SQL.
I have logs being stored in HDFS on a 10 minute window. Each 10 minute
window could have as many as 10 files with random names of 2GB each.
Now, I want to run some analysis on these files. These files are parquet
files.
I am trying to run Spark SQL queries on them.
I notice that the API only can take a single parquet File and not a
directory or a GLOB pattern where all the files can be loaded as a single
Schema RDD.

I tried doing a unionAll, but from the output of the job it looked like it
was merging the files and writing to disk ( not confirmed but from the time
it took I am assuming).

I tried insertInto, but that definitely wrote to disk and times were
comparable to unionAll operation.
Is there a way to run jobs on multiple files as if they were a single RDD.
I am not restricted to using Spark SQL, this is what I started to play
around with.
What has stopped us from creating an API that takes a GLOB pattern and
create a single RDD from all of the files inside.

Thanks
mohnish

Re: Load multiple parquet file as single RDD

[Michael Armbrust <mi...@databricks.com>]

parquetFile accepts a comma separated list of files.

Also, unionAll does not write to disk.  However, unless you are running a
recent version (compiled from master since this was added
<https://github.com/apache/spark/commit/f858f466862541c3faad76a1fa2391f1c17ec9dd>)
its missing an optimization and thus reads all the columns, instead of just
the ones required, from the disk

On Thu, Oct 2, 2014 at 6:05 PM, Mohnish Kodnani <mo...@gmail.com>
wrote:

> Hi,
> I am trying to play around with Spark and Spark SQL.
> I have logs being stored in HDFS on a 10 minute window. Each 10 minute
> window could have as many as 10 files with random names of 2GB each.
> Now, I want to run some analysis on these files. These files are parquet
> files.
> I am trying to run Spark SQL queries on them.
> I notice that the API only can take a single parquet File and not a
> directory or a GLOB pattern where all the files can be loaded as a single
> Schema RDD.
>
> I tried doing a unionAll, but from the output of the job it looked like it
> was merging the files and writing to disk ( not confirmed but from the time
> it took I am assuming).
>
> I tried insertInto, but that definitely wrote to disk and times were
> comparable to unionAll operation.
> Is there a way to run jobs on multiple files as if they were a single RDD.
> I am not restricted to using Spark SQL, this is what I started to play
> around with.
> What has stopped us from creating an API that takes a GLOB pattern and
> create a single RDD from all of the files inside.
>
> Thanks
> mohnish
>
>