You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2016/05/08 22:20:28 UTC

Parse Json in Spark

Hi,

I am working on parsing the json in spark but most of the information
available online states that  I need to have entire JSON in single line.

In my case, Json file is delivered in complex structure and not in a single
line. could anyone know how to process this in SPARK.

I used Jackson jar to process json and was able to do it when it is present
in single line. Any ideas?

Thanks,
Asmath

Re: Parse Json in Spark

Posted by KhajaAsmath Mohammed <md...@gmail.com>.

Thanks Ewan. I did the same way you explained. Thanks for your response
once again.

On Mon, May 9, 2016 at 4:21 PM, Ewan Leith <ew...@realitymine.com>
wrote:

> The simplest way is probably to use the sc.binaryFiles or
> sc.wholeTextFiles API to create an RDD containing the JSON files (maybe
> need a sc.wholeTextFiles(…).map(x => x._2) to drop off the filename column)
> then do a sqlContext.read.json(rddName)
>
>
>
> That way, you don’t need to worry about combining lines.
>
>
>
> Ewan
>
>
>
> *From:* KhajaAsmath Mohammed [mailto:mdkhajaasmath@gmail.com]
> *Sent:* 08 May 2016 23:20
> *To:* user @spark <us...@spark.apache.org>
> *Subject:* Parse Json in Spark
>
>
>
> Hi,
>
>
>
> I am working on parsing the json in spark but most of the information
> available online states that  I need to have entire JSON in single line.
>
>
>
> In my case, Json file is delivered in complex structure and not in a
> single line. could anyone know how to process this in SPARK.
>
>
>
> I used Jackson jar to process json and was able to do it when it is
> present in single line. Any ideas?
>
>
>
> Thanks,
>
> Asmath
>

RE: Parse Json in Spark

Posted by Ewan Leith <ew...@realitymine.com>.

The simplest way is probably to use the sc.binaryFiles or sc.wholeTextFiles API to create an RDD containing the JSON files (maybe need a sc.wholeTextFiles(…).map(x => x._2) to drop off the filename column) then do a sqlContext.read.json(rddName)

That way, you don’t need to worry about combining lines.

Ewan

From: KhajaAsmath Mohammed [mailto:mdkhajaasmath@gmail.com]
Sent: 08 May 2016 23:20
To: user @spark <us...@spark.apache.org>
Subject: Parse Json in Spark

Hi,

I am working on parsing the json in spark but most of the information available online states that  I need to have entire JSON in single line.

In my case, Json file is delivered in complex structure and not in a single line. could anyone know how to process this in SPARK.

I used Jackson jar to process json and was able to do it when it is present in single line. Any ideas?

Thanks,
Asmath

Re: Parse Json in Spark

Posted by Ashish Dubey <as...@gmail.com>.

This limit is due to underlying inputFormat implementation.  you can always
write your own inputFormat and then use spark newAPIHadoopFile api to pass
your inputFormat class path. You will have to place the jar file in /lib
location on all the nodes..

Ashish

On Sun, May 8, 2016 at 4:02 PM, Hyukjin Kwon <gu...@gmail.com> wrote:

>
> I remember this Jira, https://issues.apache.org/jira/browse/SPARK-7366.
> Parsing multiple lines are not supported in Json fsta source.
>
> Instead this can be done by sc.wholeTextFiles(). I found some examples
> here,
> http://searchdatascience.com/spark-adventures-1-processing-multi-line-json-files
>
> Although this reads a file as a whole record, this should work.
>
> Thanks!
> On 9 May 2016 7:20 a.m., "KhajaAsmath Mohammed" <md...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am working on parsing the json in spark but most of the information
>> available online states that  I need to have entire JSON in single line.
>>
>> In my case, Json file is delivered in complex structure and not in a
>> single line. could anyone know how to process this in SPARK.
>>
>> I used Jackson jar to process json and was able to do it when it is
>> present in single line. Any ideas?
>>
>> Thanks,
>> Asmath
>>
>

Re: Parse Json in Spark

Posted by Hyukjin Kwon <gu...@gmail.com>.

I remember this Jira, https://issues.apache.org/jira/browse/SPARK-7366.
Parsing multiple lines are not supported in Json fsta source.

Instead this can be done by sc.wholeTextFiles(). I found some examples
here,
http://searchdatascience.com/spark-adventures-1-processing-multi-line-json-files

Although this reads a file as a whole record, this should work.

Thanks!
On 9 May 2016 7:20 a.m., "KhajaAsmath Mohammed" <md...@gmail.com>
wrote:

> Hi,
>
> I am working on parsing the json in spark but most of the information
> available online states that  I need to have entire JSON in single line.
>
> In my case, Json file is delivered in complex structure and not in a
> single line. could anyone know how to process this in SPARK.
>
> I used Jackson jar to process json and was able to do it when it is
> present in single line. Any ideas?
>
> Thanks,
> Asmath
>