You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sam Flint <sa...@magnetic.com> on 2014/11/19 22:02:51 UTC
NEW to spark and sparksql
Hi,
I am new to spark. I have began to read to understand sparks RDD files
as well as SparkSQL. My question is more on how to build out the RDD files
and best practices. I have data that is broken down by hour into files on
HDFS in avro format. Do I need to create a separate RDD for each file? or
using SparkSQL a separate SchemaRDD?
I want to be able to pull lets say an entire day of data into spark and run
some analytics on it. Then possibly a week, a month, etc.
If there is documentation on this procedure or best practives for building
RDD's please point me at them.
Thanks for your time,
Sam
Re: NEW to spark and sparksql
Posted by Michael Armbrust <mi...@databricks.com>.
I believe functions like sc.textFile will also accept paths with globs for
example "/data/*/" which would read all the directories into a single RDD.
Under the covers I think it is just using Hadoop's FileInputFormat, in case
you want to google for the full list of supported syntax.
On Thu, Nov 20, 2014 at 7:27 AM, Sam Flint <sa...@magnetic.com> wrote:
> So you are saying to query an entire day of data I would need to create
> one RDD for every hour and then union them into one RDD. After I have the
> one RDD I would be able to query for a=2 throughout the entire day.
> Please correct me if I am wrong.
>
> Thanks
>
> On Wed, Nov 19, 2014 at 5:53 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> I would use just textFile unless you actually need a guarantee that you
>> will be seeing a whole file at time (textFile splits on new lines).
>>
>> RDDs are immutable, so you cannot add data to them. You can however
>> union two RDDs, returning a new RDD that contains all the data.
>>
>> On Wed, Nov 19, 2014 at 2:46 PM, Sam Flint <sa...@magnetic.com>
>> wrote:
>>
>>> Michael,
>>> Thanks for your help. I found a wholeTextFiles() that I can use to
>>> import all files in a directory. I believe this would be the case if all
>>> the files existed in the same directory. Currently the files come in by
>>> the hour and are in a format somewhat like this ../2014/10/01/00/filename
>>> and there is one file per hour.
>>>
>>> Do I create an RDD and add to it? Is that possible? My example query
>>> would be select count(*) from (entire day RDD) where a=2. "a" would exist
>>> in all files multiple times with multiple values.
>>>
>>> I don't see in any documentation how to import a file create an RDD then
>>> import another file into that RDD. kinda like in mysql when you create a
>>> table import data then import more data. This may be my ignorance because
>>> I am not that familiar with spark, but I would expect to import data into a
>>> single RDD before performing analytics on it.
>>>
>>> Thank you for your time and help on this.
>>>
>>>
>>> P.S. I am using python if that makes a difference.
>>>
>>> On Wed, Nov 19, 2014 at 4:45 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> In general you should be able to read full directories of files as a
>>>> single RDD/SchemaRDD. For documentation I'd suggest the programming
>>>> guides:
>>>>
>>>> http://spark.apache.org/docs/latest/quick-start.html
>>>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>>>
>>>> For Avro in particular, I have been working on a library for Spark
>>>> SQL. Its very early code, but you can find it here:
>>>> https://github.com/databricks/spark-avro
>>>>
>>>> Bug reports welcome!
>>>>
>>>> Michael
>>>>
>>>> On Wed, Nov 19, 2014 at 1:02 PM, Sam Flint <sa...@magnetic.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am new to spark. I have began to read to understand sparks RDD
>>>>> files as well as SparkSQL. My question is more on how to build out the RDD
>>>>> files and best practices. I have data that is broken down by hour into
>>>>> files on HDFS in avro format. Do I need to create a separate RDD for each
>>>>> file? or using SparkSQL a separate SchemaRDD?
>>>>>
>>>>> I want to be able to pull lets say an entire day of data into spark
>>>>> and run some analytics on it. Then possibly a week, a month, etc.
>>>>>
>>>>>
>>>>> If there is documentation on this procedure or best practives for
>>>>> building RDD's please point me at them.
>>>>>
>>>>> Thanks for your time,
>>>>> Sam
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *MAGNE**+**I**C*
>>>
>>> *Sam Flint* | *Lead Developer, Data Analytics*
>>>
>>>
>>>
>>
>
>
> --
>
> *MAGNE**+**I**C*
>
> *Sam Flint* | *Lead Developer, Data Analytics*
>
>
>
Re: NEW to spark and sparksql
Posted by Sam Flint <sa...@magnetic.com>.
So you are saying to query an entire day of data I would need to create one
RDD for every hour and then union them into one RDD. After I have the one
RDD I would be able to query for a=2 throughout the entire day. Please
correct me if I am wrong.
Thanks
On Wed, Nov 19, 2014 at 5:53 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> I would use just textFile unless you actually need a guarantee that you
> will be seeing a whole file at time (textFile splits on new lines).
>
> RDDs are immutable, so you cannot add data to them. You can however union
> two RDDs, returning a new RDD that contains all the data.
>
> On Wed, Nov 19, 2014 at 2:46 PM, Sam Flint <sa...@magnetic.com> wrote:
>
>> Michael,
>> Thanks for your help. I found a wholeTextFiles() that I can use to
>> import all files in a directory. I believe this would be the case if all
>> the files existed in the same directory. Currently the files come in by
>> the hour and are in a format somewhat like this ../2014/10/01/00/filename
>> and there is one file per hour.
>>
>> Do I create an RDD and add to it? Is that possible? My example query
>> would be select count(*) from (entire day RDD) where a=2. "a" would exist
>> in all files multiple times with multiple values.
>>
>> I don't see in any documentation how to import a file create an RDD then
>> import another file into that RDD. kinda like in mysql when you create a
>> table import data then import more data. This may be my ignorance because
>> I am not that familiar with spark, but I would expect to import data into a
>> single RDD before performing analytics on it.
>>
>> Thank you for your time and help on this.
>>
>>
>> P.S. I am using python if that makes a difference.
>>
>> On Wed, Nov 19, 2014 at 4:45 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> In general you should be able to read full directories of files as a
>>> single RDD/SchemaRDD. For documentation I'd suggest the programming
>>> guides:
>>>
>>> http://spark.apache.org/docs/latest/quick-start.html
>>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>>
>>> For Avro in particular, I have been working on a library for Spark SQL.
>>> Its very early code, but you can find it here:
>>> https://github.com/databricks/spark-avro
>>>
>>> Bug reports welcome!
>>>
>>> Michael
>>>
>>> On Wed, Nov 19, 2014 at 1:02 PM, Sam Flint <sa...@magnetic.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to spark. I have began to read to understand sparks RDD
>>>> files as well as SparkSQL. My question is more on how to build out the RDD
>>>> files and best practices. I have data that is broken down by hour into
>>>> files on HDFS in avro format. Do I need to create a separate RDD for each
>>>> file? or using SparkSQL a separate SchemaRDD?
>>>>
>>>> I want to be able to pull lets say an entire day of data into spark and
>>>> run some analytics on it. Then possibly a week, a month, etc.
>>>>
>>>>
>>>> If there is documentation on this procedure or best practives for
>>>> building RDD's please point me at them.
>>>>
>>>> Thanks for your time,
>>>> Sam
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> *MAGNE**+**I**C*
>>
>> *Sam Flint* | *Lead Developer, Data Analytics*
>>
>>
>>
>
--
*MAGNE**+**I**C*
*Sam Flint* | *Lead Developer, Data Analytics*
Re: NEW to spark and sparksql
Posted by Michael Armbrust <mi...@databricks.com>.
I would use just textFile unless you actually need a guarantee that you
will be seeing a whole file at time (textFile splits on new lines).
RDDs are immutable, so you cannot add data to them. You can however union
two RDDs, returning a new RDD that contains all the data.
On Wed, Nov 19, 2014 at 2:46 PM, Sam Flint <sa...@magnetic.com> wrote:
> Michael,
> Thanks for your help. I found a wholeTextFiles() that I can use to
> import all files in a directory. I believe this would be the case if all
> the files existed in the same directory. Currently the files come in by
> the hour and are in a format somewhat like this ../2014/10/01/00/filename
> and there is one file per hour.
>
> Do I create an RDD and add to it? Is that possible? My example query
> would be select count(*) from (entire day RDD) where a=2. "a" would exist
> in all files multiple times with multiple values.
>
> I don't see in any documentation how to import a file create an RDD then
> import another file into that RDD. kinda like in mysql when you create a
> table import data then import more data. This may be my ignorance because
> I am not that familiar with spark, but I would expect to import data into a
> single RDD before performing analytics on it.
>
> Thank you for your time and help on this.
>
>
> P.S. I am using python if that makes a difference.
>
> On Wed, Nov 19, 2014 at 4:45 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> In general you should be able to read full directories of files as a
>> single RDD/SchemaRDD. For documentation I'd suggest the programming
>> guides:
>>
>> http://spark.apache.org/docs/latest/quick-start.html
>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>
>> For Avro in particular, I have been working on a library for Spark SQL.
>> Its very early code, but you can find it here:
>> https://github.com/databricks/spark-avro
>>
>> Bug reports welcome!
>>
>> Michael
>>
>> On Wed, Nov 19, 2014 at 1:02 PM, Sam Flint <sa...@magnetic.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am new to spark. I have began to read to understand sparks RDD
>>> files as well as SparkSQL. My question is more on how to build out the RDD
>>> files and best practices. I have data that is broken down by hour into
>>> files on HDFS in avro format. Do I need to create a separate RDD for each
>>> file? or using SparkSQL a separate SchemaRDD?
>>>
>>> I want to be able to pull lets say an entire day of data into spark and
>>> run some analytics on it. Then possibly a week, a month, etc.
>>>
>>>
>>> If there is documentation on this procedure or best practives for
>>> building RDD's please point me at them.
>>>
>>> Thanks for your time,
>>> Sam
>>>
>>>
>>>
>>>
>>
>
>
> --
>
> *MAGNE**+**I**C*
>
> *Sam Flint* | *Lead Developer, Data Analytics*
>
>
>
Re: NEW to spark and sparksql
Posted by Michael Armbrust <mi...@databricks.com>.
In general you should be able to read full directories of files as a single
RDD/SchemaRDD. For documentation I'd suggest the programming guides:
http://spark.apache.org/docs/latest/quick-start.html
http://spark.apache.org/docs/latest/sql-programming-guide.html
For Avro in particular, I have been working on a library for Spark SQL.
Its very early code, but you can find it here:
https://github.com/databricks/spark-avro
Bug reports welcome!
Michael
On Wed, Nov 19, 2014 at 1:02 PM, Sam Flint <sa...@magnetic.com> wrote:
> Hi,
>
> I am new to spark. I have began to read to understand sparks RDD
> files as well as SparkSQL. My question is more on how to build out the RDD
> files and best practices. I have data that is broken down by hour into
> files on HDFS in avro format. Do I need to create a separate RDD for each
> file? or using SparkSQL a separate SchemaRDD?
>
> I want to be able to pull lets say an entire day of data into spark and
> run some analytics on it. Then possibly a week, a month, etc.
>
>
> If there is documentation on this procedure or best practives for building
> RDD's please point me at them.
>
> Thanks for your time,
> Sam
>
>
>
>