You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jerry Lam <ch...@gmail.com> on 2014/12/17 17:21:40 UTC

Spark SQL 1.1.1 reading LZO compressed json files

Hi spark users,

Do you know how to read json files using Spark SQL that are LZO compressed?

I'm looking into sqlContext.jsonFile but I don't know how to configure it
to read lzo files.

Best Regards,

Jerry

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Ted Yu <yu...@gmail.com>.
In SQLContext:
  def jsonFile(path: String, samplingRatio: Double): SchemaRDD = {
    val json = sparkContext.textFile(path)
    jsonRDD(json, samplingRatio)
  }
Looks like jsonFile() can be enhanced with call to
sparkContext.newAPIHadoopFile()
with proper input file format.

Cheers

On Wed, Dec 17, 2014 at 8:33 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi Ted,
>
> Thanks for your help.
> I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
> couldn't do the same for sqlContext because sqlContext.josnFile does not
> provide ways to configure the input file format. Do you know if there are
> some APIs to do that?
>
> Best Regards,
>
> Jerry
>
> On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
>> which references https://issues.apache.org/jira/browse/SPARK-2394
>>
>> Cheers
>>
>> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>> Hi spark users,
>>>
>>> Do you know how to read json files using Spark SQL that are LZO
>>> compressed?
>>>
>>> I'm looking into sqlContext.jsonFile but I don't know how to configure
>>> it to read lzo files.
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Michael Armbrust <mi...@databricks.com>.
To be a little more clear jsonRDD and jsonFile use the same implementation
underneath.  jsonFile is just a connivence method that does
jsonRDD(sc.textFile(...))

On Wed, Dec 17, 2014 at 11:37 AM, Michael Armbrust <mi...@databricks.com>
wrote:
>
> The first pass is inferring the schema of the JSON data.  If you already
> know the schema you can skip this pass by specifying the schema as the
> second parameter to jsonRDD.
>
> On Wed, Dec 17, 2014 at 10:59 AM, Jerry Lam <ch...@gmail.com> wrote:
>>
>> Hi Michael,
>>
>> This is what I did. I was thinking if there is a more efficient way to
>> accomplish this.
>>
>> I was doing a very simple benchmark: Convert lzo compressed json files to
>> parquet files using SparkSQL vs. Hadoop MR.
>>
>> Spark SQL seems to require 2 stages to accomplish this task:
>> Stage 1: read the lzo files using newAPIHadoopFile with
>> LzoTextInputFormat and then convert it to JsonRDD
>> Stage 2: saveAsParquetFile from the JsonRDD
>>
>> In Hadoop, it takes 1 step, a map-only job to read the data and then
>> output the json to the parquet file (I'm using elephant bird LzoJsonLoader
>> to load the files)
>>
>> In some scenarios, Hadoop is faster because it is saving one stage. Did I
>> do something wrong?
>>
>> Best Regards,
>>
>> Jerry
>>
>>
>> On Wed, Dec 17, 2014 at 1:29 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>>
>>> You can create an RDD[String] using whatever method and pass that to
>>> jsonRDD.
>>>
>>> On Wed, Dec 17, 2014 at 8:33 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>>
>>>> Hi Ted,
>>>>
>>>> Thanks for your help.
>>>> I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
>>>> couldn't do the same for sqlContext because sqlContext.josnFile does not
>>>> provide ways to configure the input file format. Do you know if there are
>>>> some APIs to do that?
>>>>
>>>> Best Regards,
>>>>
>>>> Jerry
>>>>
>>>> On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
>>>>> which references https://issues.apache.org/jira/browse/SPARK-2394
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi spark users,
>>>>>>
>>>>>> Do you know how to read json files using Spark SQL that are LZO
>>>>>> compressed?
>>>>>>
>>>>>> I'm looking into sqlContext.jsonFile but I don't know how to
>>>>>> configure it to read lzo files.
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Jerry
>>>>>>
>>>>>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Michael Armbrust <mi...@databricks.com>.
The first pass is inferring the schema of the JSON data.  If you already
know the schema you can skip this pass by specifying the schema as the
second parameter to jsonRDD.

On Wed, Dec 17, 2014 at 10:59 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi Michael,
>
> This is what I did. I was thinking if there is a more efficient way to
> accomplish this.
>
> I was doing a very simple benchmark: Convert lzo compressed json files to
> parquet files using SparkSQL vs. Hadoop MR.
>
> Spark SQL seems to require 2 stages to accomplish this task:
> Stage 1: read the lzo files using newAPIHadoopFile with LzoTextInputFormat
> and then convert it to JsonRDD
> Stage 2: saveAsParquetFile from the JsonRDD
>
> In Hadoop, it takes 1 step, a map-only job to read the data and then
> output the json to the parquet file (I'm using elephant bird LzoJsonLoader
> to load the files)
>
> In some scenarios, Hadoop is faster because it is saving one stage. Did I
> do something wrong?
>
> Best Regards,
>
> Jerry
>
>
> On Wed, Dec 17, 2014 at 1:29 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>>
>> You can create an RDD[String] using whatever method and pass that to
>> jsonRDD.
>>
>> On Wed, Dec 17, 2014 at 8:33 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>> Hi Ted,
>>>
>>> Thanks for your help.
>>> I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
>>> couldn't do the same for sqlContext because sqlContext.josnFile does not
>>> provide ways to configure the input file format. Do you know if there are
>>> some APIs to do that?
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>> On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
>>>> which references https://issues.apache.org/jira/browse/SPARK-2394
>>>>
>>>> Cheers
>>>>
>>>> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi spark users,
>>>>>
>>>>> Do you know how to read json files using Spark SQL that are LZO
>>>>> compressed?
>>>>>
>>>>> I'm looking into sqlContext.jsonFile but I don't know how to configure
>>>>> it to read lzo files.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Jerry
>>>>>
>>>>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Jerry Lam <ch...@gmail.com>.
Hi Michael,

This is what I did. I was thinking if there is a more efficient way to
accomplish this.

I was doing a very simple benchmark: Convert lzo compressed json files to
parquet files using SparkSQL vs. Hadoop MR.

Spark SQL seems to require 2 stages to accomplish this task:
Stage 1: read the lzo files using newAPIHadoopFile with LzoTextInputFormat
and then convert it to JsonRDD
Stage 2: saveAsParquetFile from the JsonRDD

In Hadoop, it takes 1 step, a map-only job to read the data and then output
the json to the parquet file (I'm using elephant bird LzoJsonLoader to load
the files)

In some scenarios, Hadoop is faster because it is saving one stage. Did I
do something wrong?

Best Regards,

Jerry


On Wed, Dec 17, 2014 at 1:29 PM, Michael Armbrust <mi...@databricks.com>
wrote:
>
> You can create an RDD[String] using whatever method and pass that to
> jsonRDD.
>
> On Wed, Dec 17, 2014 at 8:33 AM, Jerry Lam <ch...@gmail.com> wrote:
>>
>> Hi Ted,
>>
>> Thanks for your help.
>> I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
>> couldn't do the same for sqlContext because sqlContext.josnFile does not
>> provide ways to configure the input file format. Do you know if there are
>> some APIs to do that?
>>
>> Best Regards,
>>
>> Jerry
>>
>> On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
>>> which references https://issues.apache.org/jira/browse/SPARK-2394
>>>
>>> Cheers
>>>
>>> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>>
>>>> Hi spark users,
>>>>
>>>> Do you know how to read json files using Spark SQL that are LZO
>>>> compressed?
>>>>
>>>> I'm looking into sqlContext.jsonFile but I don't know how to configure
>>>> it to read lzo files.
>>>>
>>>> Best Regards,
>>>>
>>>> Jerry
>>>>
>>>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Michael Armbrust <mi...@databricks.com>.
You can create an RDD[String] using whatever method and pass that to
jsonRDD.

On Wed, Dec 17, 2014 at 8:33 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi Ted,
>
> Thanks for your help.
> I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
> couldn't do the same for sqlContext because sqlContext.josnFile does not
> provide ways to configure the input file format. Do you know if there are
> some APIs to do that?
>
> Best Regards,
>
> Jerry
>
> On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
>> which references https://issues.apache.org/jira/browse/SPARK-2394
>>
>> Cheers
>>
>> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>> Hi spark users,
>>>
>>> Do you know how to read json files using Spark SQL that are LZO
>>> compressed?
>>>
>>> I'm looking into sqlContext.jsonFile but I don't know how to configure
>>> it to read lzo files.
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Jerry Lam <ch...@gmail.com>.
Hi Ted,

Thanks for your help.
I'm able to read lzo files using sparkContext.newAPIHadoopFile but I
couldn't do the same for sqlContext because sqlContext.josnFile does not
provide ways to configure the input file format. Do you know if there are
some APIs to do that?

Best Regards,

Jerry

On Wed, Dec 17, 2014 at 11:27 AM, Ted Yu <yu...@gmail.com> wrote:
>
> See this thread: http://search-hadoop.com/m/JW1q5HAuFv
> which references https://issues.apache.org/jira/browse/SPARK-2394
>
> Cheers
>
> On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com> wrote:
>>
>> Hi spark users,
>>
>> Do you know how to read json files using Spark SQL that are LZO
>> compressed?
>>
>> I'm looking into sqlContext.jsonFile but I don't know how to configure it
>> to read lzo files.
>>
>> Best Regards,
>>
>> Jerry
>>
>

Re: Spark SQL 1.1.1 reading LZO compressed json files

Posted by Ted Yu <yu...@gmail.com>.
See this thread: http://search-hadoop.com/m/JW1q5HAuFv
which references https://issues.apache.org/jira/browse/SPARK-2394

Cheers

On Wed, Dec 17, 2014 at 8:21 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi spark users,
>
> Do you know how to read json files using Spark SQL that are LZO compressed?
>
> I'm looking into sqlContext.jsonFile but I don't know how to configure it
> to read lzo files.
>
> Best Regards,
>
> Jerry
>