You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Udit Mehta <um...@groupon.com> on 2015/08/18 22:12:18 UTC

Json Serde used by Spark Sql

Hi,

I was wondering what json serde does spark sql use. I created a JsonRDD out
of a json file and then registered it as a temp table to query. I can then
query the table using dot notation for nested structs/arrays. I was
wondering how does spark sql deserialize the json data based on the query.

Thanks in advance,
Udit

Re: Json Serde used by Spark Sql

Posted by Michael Armbrust <mi...@databricks.com>.

Under the covers we use Jackson's Streaming API as of Spark 1.4.

On Tue, Aug 18, 2015 at 1:12 PM, Udit Mehta <um...@groupon.com> wrote:

> Hi,
>
> I was wondering what json serde does spark sql use. I created a JsonRDD
> out of a json file and then registered it as a temp table to query. I can
> then query the table using dot notation for nested structs/arrays. I was
> wondering how does spark sql deserialize the json data based on the query.
>
> Thanks in advance,
> Udit
>