You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2016/11/18 07:42:29 UTC
How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Hi All,
I would like to flatten JSON blobs into a Data Frame using Spark/Spark SQl
inside Spark-Shell.
val df = spark.sql("select body from test limit 3"); // body is a json
encoded blob column
val df2 = df.select(df("body").cast(StringType).as("body"))
when I do
df2.show // shows the 3 rows
body
------------------------------------
{"k1": "v1", "k2": "v2" }
{"k3": "v3"}
{"k4": "v4", "k5": "v5", "k6": "v6"}
-------------------------------------
Now say I have billion of these rows/records but at most there will 5
different json schemas for all billion rows. Now how do I flatten such that
I get a data frame in the format below? Should I use df.forEach or
df.forEachPartition or df.explode or df.flatMap? How do I make sure I am
not creating a billion data frames and trying to union all of them or
something more inefficient. It will be great if I could see a code sample.
Also since this might have Nil I wonder if they would take up any space? I
initially tried it with UDF's and I keep getting some exception saying
"Any" Type is not supported and so on. So an example on how to do this
would help!
"K1" | "K2" | "K3" | "K4" | "K5" | "K6"
---------------------------------------
"V1" | "V2" |
| "V3" |
| "V4" | "V5" | "V6"
Needless to say, V1, V2, V3...V6 can be any type (String, boolean, integer,
Map...). Also open to any new ideas.
Thanks!
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by Hyukjin Kwon <gu...@gmail.com>.
Hi Kant,
Ah, I thought you wanted to find the workaround to so it.
Then wouldn't this be easily able to reach the same goal with the
workaround without new such API?
Thanks.
On 6 Dec 2016 4:11 a.m., "kant kodali" <ka...@gmail.com> wrote:
> Hi Kwon,
>
> Thanks for this but Isn't this what Michael suggested?
>
> Thanks,
> kant
>
> On Mon, Dec 5, 2016 at 4:45 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Hi Kant,
>>
>> How about doing something like this?
>>
>> import org.apache.spark.sql.functions._
>>
>> // val df2 = df.select(df("body").cast(StringType).as("body"))
>> val df2 = Seq("""{"a": 1}""").toDF("body")
>> val schema = spark.read.json(df2.as[String].rdd).schema
>> df2.select(from_json(col("body"), schema)).show()
>>
>>
>>
>> 2016-12-05 19:51 GMT+09:00 kant kodali <ka...@gmail.com>:
>>
>>> Hi Michael,
>>>
>>> " Personally, I usually take a small sample of data and use schema
>>> inference on that. I then hardcode that schema into my program. This
>>> makes your spark jobs much faster and removes the possibility of the schema
>>> changing underneath the covers."
>>>
>>> This may or may not work for us. Not all rows have the same schema. The
>>> number of distinct schemas we have now may be small but going forward this
>>> can go to any number moreover a distinct call can lead to a table scan
>>> which can be billions of rows for us.
>>>
>>> I also would agree to keep the API consistent than making an exception
>>> however I wonder if it make sense to provide an action call to infer the
>>> schema which would return a new dataframe after the action call finishes
>>> (after schema inference)? For example, something like below ?
>>>
>>> val inferedDF = df.inferSchema(col1);
>>>
>>> Thanks,
>>>
>>>
>>>
>>>
>>> On Mon, Nov 28, 2016 at 6:12 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> You could open up a JIRA to add a version of from_json that supports
>>>> schema inference, but unfortunately that would not be super easy to
>>>> implement. In particular, it would introduce a weird case where only this
>>>> specific function would block for a long time while we infer the schema
>>>> (instead of waiting for an action). This blocking would be kind of odd for
>>>> a call like df.select(...). If there is enough interest, though, we
>>>> should still do it.
>>>>
>>>> To give a little more detail, your version of the code is actually
>>>> doing two passes over the data: one to infer the schema and a second for
>>>> whatever processing you are asking it to do. We have to know the schema at
>>>> each step of DataFrame construction, so we'd have to do this even before
>>>> you called an action.
>>>>
>>>> Personally, I usually take a small sample of data and use schema
>>>> inference on that. I then hardcode that schema into my program. This
>>>> makes your spark jobs much faster and removes the possibility of the schema
>>>> changing underneath the covers.
>>>>
>>>> Here's some code I use to build the static schema code automatically
>>>> <https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1128172975083446/2840265927289860/latest.html>
>>>> .
>>>>
>>>> Would that work for you? If not, why not?
>>>>
>>>> On Wed, Nov 23, 2016 at 2:48 AM, kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Looks like all from_json functions will require me to pass schema and
>>>>> that can be little tricky for us but the code below doesn't require me to
>>>>> pass schema at all.
>>>>>
>>>>> import org.apache.spark.sql._
>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>> spark.read.json(rdd).show()
>>>>>
>>>>>
>>>>> On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <
>>>>> michael@databricks.com> wrote:
>>>>>
>>>>>> The first release candidate should be coming out this week. You can
>>>>>> subscribe to the dev list if you want to follow the release schedule.
>>>>>>
>>>>>> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> I only see spark 2.0.2 which is what I am using currently. Any idea
>>>>>>> on when 2.1 will be released?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> kant
>>>>>>>
>>>>>>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <
>>>>>>> michael@databricks.com> wrote:
>>>>>>>
>>>>>>>> In Spark 2.1 we've added a from_json
>>>>>>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>>>>>>> function that I think will do what you want.
>>>>>>>>
>>>>>>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This seem to work
>>>>>>>>>
>>>>>>>>> import org.apache.spark.sql._
>>>>>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>>>>>> spark.read.json(rdd).show()
>>>>>>>>>
>>>>>>>>> However I wonder if this any inefficiency here ? since I have to
>>>>>>>>> apply this function for billion rows.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by kant kodali <ka...@gmail.com>.
Hi Kwon,
Thanks for this but Isn't this what Michael suggested?
Thanks,
kant
On Mon, Dec 5, 2016 at 4:45 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
> Hi Kant,
>
> How about doing something like this?
>
> import org.apache.spark.sql.functions._
>
> // val df2 = df.select(df("body").cast(StringType).as("body"))
> val df2 = Seq("""{"a": 1}""").toDF("body")
> val schema = spark.read.json(df2.as[String].rdd).schema
> df2.select(from_json(col("body"), schema)).show()
>
>
>
> 2016-12-05 19:51 GMT+09:00 kant kodali <ka...@gmail.com>:
>
>> Hi Michael,
>>
>> " Personally, I usually take a small sample of data and use schema
>> inference on that. I then hardcode that schema into my program. This
>> makes your spark jobs much faster and removes the possibility of the schema
>> changing underneath the covers."
>>
>> This may or may not work for us. Not all rows have the same schema. The
>> number of distinct schemas we have now may be small but going forward this
>> can go to any number moreover a distinct call can lead to a table scan
>> which can be billions of rows for us.
>>
>> I also would agree to keep the API consistent than making an exception
>> however I wonder if it make sense to provide an action call to infer the
>> schema which would return a new dataframe after the action call finishes
>> (after schema inference)? For example, something like below ?
>>
>> val inferedDF = df.inferSchema(col1);
>>
>> Thanks,
>>
>>
>>
>>
>> On Mon, Nov 28, 2016 at 6:12 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> You could open up a JIRA to add a version of from_json that supports
>>> schema inference, but unfortunately that would not be super easy to
>>> implement. In particular, it would introduce a weird case where only this
>>> specific function would block for a long time while we infer the schema
>>> (instead of waiting for an action). This blocking would be kind of odd for
>>> a call like df.select(...). If there is enough interest, though, we
>>> should still do it.
>>>
>>> To give a little more detail, your version of the code is actually doing
>>> two passes over the data: one to infer the schema and a second for whatever
>>> processing you are asking it to do. We have to know the schema at each
>>> step of DataFrame construction, so we'd have to do this even before you
>>> called an action.
>>>
>>> Personally, I usually take a small sample of data and use schema
>>> inference on that. I then hardcode that schema into my program. This
>>> makes your spark jobs much faster and removes the possibility of the schema
>>> changing underneath the covers.
>>>
>>> Here's some code I use to build the static schema code automatically
>>> <https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1128172975083446/2840265927289860/latest.html>
>>> .
>>>
>>> Would that work for you? If not, why not?
>>>
>>> On Wed, Nov 23, 2016 at 2:48 AM, kant kodali <ka...@gmail.com> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> Looks like all from_json functions will require me to pass schema and
>>>> that can be little tricky for us but the code below doesn't require me to
>>>> pass schema at all.
>>>>
>>>> import org.apache.spark.sql._
>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>> spark.read.json(rdd).show()
>>>>
>>>>
>>>> On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>>> The first release candidate should be coming out this week. You can
>>>>> subscribe to the dev list if you want to follow the release schedule.
>>>>>
>>>>> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> I only see spark 2.0.2 which is what I am using currently. Any idea
>>>>>> on when 2.1 will be released?
>>>>>>
>>>>>> Thanks,
>>>>>> kant
>>>>>>
>>>>>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <
>>>>>> michael@databricks.com> wrote:
>>>>>>
>>>>>>> In Spark 2.1 we've added a from_json
>>>>>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>>>>>> function that I think will do what you want.
>>>>>>>
>>>>>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> This seem to work
>>>>>>>>
>>>>>>>> import org.apache.spark.sql._
>>>>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>>>>> spark.read.json(rdd).show()
>>>>>>>>
>>>>>>>> However I wonder if this any inefficiency here ? since I have to
>>>>>>>> apply this function for billion rows.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by Hyukjin Kwon <gu...@gmail.com>.
Hi Kant,
How about doing something like this?
import org.apache.spark.sql.functions._
// val df2 = df.select(df("body").cast(StringType).as("body"))
val df2 = Seq("""{"a": 1}""").toDF("body")
val schema = spark.read.json(df2.as[String].rdd).schema
df2.select(from_json(col("body"), schema)).show()
2016-12-05 19:51 GMT+09:00 kant kodali <ka...@gmail.com>:
> Hi Michael,
>
> " Personally, I usually take a small sample of data and use schema
> inference on that. I then hardcode that schema into my program. This
> makes your spark jobs much faster and removes the possibility of the schema
> changing underneath the covers."
>
> This may or may not work for us. Not all rows have the same schema. The
> number of distinct schemas we have now may be small but going forward this
> can go to any number moreover a distinct call can lead to a table scan
> which can be billions of rows for us.
>
> I also would agree to keep the API consistent than making an exception
> however I wonder if it make sense to provide an action call to infer the
> schema which would return a new dataframe after the action call finishes
> (after schema inference)? For example, something like below ?
>
> val inferedDF = df.inferSchema(col1);
>
> Thanks,
>
>
>
>
> On Mon, Nov 28, 2016 at 6:12 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> You could open up a JIRA to add a version of from_json that supports
>> schema inference, but unfortunately that would not be super easy to
>> implement. In particular, it would introduce a weird case where only this
>> specific function would block for a long time while we infer the schema
>> (instead of waiting for an action). This blocking would be kind of odd for
>> a call like df.select(...). If there is enough interest, though, we
>> should still do it.
>>
>> To give a little more detail, your version of the code is actually doing
>> two passes over the data: one to infer the schema and a second for whatever
>> processing you are asking it to do. We have to know the schema at each
>> step of DataFrame construction, so we'd have to do this even before you
>> called an action.
>>
>> Personally, I usually take a small sample of data and use schema
>> inference on that. I then hardcode that schema into my program. This
>> makes your spark jobs much faster and removes the possibility of the schema
>> changing underneath the covers.
>>
>> Here's some code I use to build the static schema code automatically
>> <https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1128172975083446/2840265927289860/latest.html>
>> .
>>
>> Would that work for you? If not, why not?
>>
>> On Wed, Nov 23, 2016 at 2:48 AM, kant kodali <ka...@gmail.com> wrote:
>>
>>> Hi Michael,
>>>
>>> Looks like all from_json functions will require me to pass schema and
>>> that can be little tricky for us but the code below doesn't require me to
>>> pass schema at all.
>>>
>>> import org.apache.spark.sql._
>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>> spark.read.json(rdd).show()
>>>
>>>
>>> On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> The first release candidate should be coming out this week. You can
>>>> subscribe to the dev list if you want to follow the release schedule.
>>>>
>>>> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Michael,
>>>>>
>>>>> I only see spark 2.0.2 which is what I am using currently. Any idea on
>>>>> when 2.1 will be released?
>>>>>
>>>>> Thanks,
>>>>> kant
>>>>>
>>>>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <
>>>>> michael@databricks.com> wrote:
>>>>>
>>>>>> In Spark 2.1 we've added a from_json
>>>>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>>>>> function that I think will do what you want.
>>>>>>
>>>>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> This seem to work
>>>>>>>
>>>>>>> import org.apache.spark.sql._
>>>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>>>> spark.read.json(rdd).show()
>>>>>>>
>>>>>>> However I wonder if this any inefficiency here ? since I have to
>>>>>>> apply this function for billion rows.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by kant kodali <ka...@gmail.com>.
Hi Michael,
" Personally, I usually take a small sample of data and use schema
inference on that. I then hardcode that schema into my program. This
makes your spark jobs much faster and removes the possibility of the schema
changing underneath the covers."
This may or may not work for us. Not all rows have the same schema. The
number of distinct schemas we have now may be small but going forward this
can go to any number moreover a distinct call can lead to a table scan
which can be billions of rows for us.
I also would agree to keep the API consistent than making an exception
however I wonder if it make sense to provide an action call to infer the
schema which would return a new dataframe after the action call finishes
(after schema inference)? For example, something like below ?
val inferedDF = df.inferSchema(col1);
Thanks,
On Mon, Nov 28, 2016 at 6:12 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> You could open up a JIRA to add a version of from_json that supports
> schema inference, but unfortunately that would not be super easy to
> implement. In particular, it would introduce a weird case where only this
> specific function would block for a long time while we infer the schema
> (instead of waiting for an action). This blocking would be kind of odd for
> a call like df.select(...). If there is enough interest, though, we
> should still do it.
>
> To give a little more detail, your version of the code is actually doing
> two passes over the data: one to infer the schema and a second for whatever
> processing you are asking it to do. We have to know the schema at each
> step of DataFrame construction, so we'd have to do this even before you
> called an action.
>
> Personally, I usually take a small sample of data and use schema inference
> on that. I then hardcode that schema into my program. This makes your
> spark jobs much faster and removes the possibility of the schema changing
> underneath the covers.
>
> Here's some code I use to build the static schema code automatically
> <https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1128172975083446/2840265927289860/latest.html>
> .
>
> Would that work for you? If not, why not?
>
> On Wed, Nov 23, 2016 at 2:48 AM, kant kodali <ka...@gmail.com> wrote:
>
>> Hi Michael,
>>
>> Looks like all from_json functions will require me to pass schema and
>> that can be little tricky for us but the code below doesn't require me to
>> pass schema at all.
>>
>> import org.apache.spark.sql._
>> val rdd = df2.rdd.map { case Row(j: String) => j }
>> spark.read.json(rdd).show()
>>
>>
>> On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> The first release candidate should be coming out this week. You can
>>> subscribe to the dev list if you want to follow the release schedule.
>>>
>>> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> I only see spark 2.0.2 which is what I am using currently. Any idea on
>>>> when 2.1 will be released?
>>>>
>>>> Thanks,
>>>> kant
>>>>
>>>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>>> In Spark 2.1 we've added a from_json
>>>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>>>> function that I think will do what you want.
>>>>>
>>>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> This seem to work
>>>>>>
>>>>>> import org.apache.spark.sql._
>>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>>> spark.read.json(rdd).show()
>>>>>>
>>>>>> However I wonder if this any inefficiency here ? since I have to
>>>>>> apply this function for billion rows.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by Michael Armbrust <mi...@databricks.com>.
You could open up a JIRA to add a version of from_json that supports schema
inference, but unfortunately that would not be super easy to implement. In
particular, it would introduce a weird case where only this specific
function would block for a long time while we infer the schema (instead of
waiting for an action). This blocking would be kind of odd for a call like
df.select(...). If there is enough interest, though, we should still do it.
To give a little more detail, your version of the code is actually doing
two passes over the data: one to infer the schema and a second for whatever
processing you are asking it to do. We have to know the schema at each
step of DataFrame construction, so we'd have to do this even before you
called an action.
Personally, I usually take a small sample of data and use schema inference
on that. I then hardcode that schema into my program. This makes your
spark jobs much faster and removes the possibility of the schema changing
underneath the covers.
Here's some code I use to build the static schema code automatically
<https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1128172975083446/2840265927289860/latest.html>
.
Would that work for you? If not, why not?
On Wed, Nov 23, 2016 at 2:48 AM, kant kodali <ka...@gmail.com> wrote:
> Hi Michael,
>
> Looks like all from_json functions will require me to pass schema and that
> can be little tricky for us but the code below doesn't require me to pass
> schema at all.
>
> import org.apache.spark.sql._
> val rdd = df2.rdd.map { case Row(j: String) => j }
> spark.read.json(rdd).show()
>
>
> On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> The first release candidate should be coming out this week. You can
>> subscribe to the dev list if you want to follow the release schedule.
>>
>> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com> wrote:
>>
>>> Hi Michael,
>>>
>>> I only see spark 2.0.2 which is what I am using currently. Any idea on
>>> when 2.1 will be released?
>>>
>>> Thanks,
>>> kant
>>>
>>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> In Spark 2.1 we've added a from_json
>>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>>> function that I think will do what you want.
>>>>
>>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> This seem to work
>>>>>
>>>>> import org.apache.spark.sql._
>>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>>> spark.read.json(rdd).show()
>>>>>
>>>>> However I wonder if this any inefficiency here ? since I have to apply
>>>>> this function for billion rows.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by kant kodali <ka...@gmail.com>.
Hi Michael,
Looks like all from_json functions will require me to pass schema and that
can be little tricky for us but the code below doesn't require me to pass
schema at all.
import org.apache.spark.sql._
val rdd = df2.rdd.map { case Row(j: String) => j }
spark.read.json(rdd).show()
On Tue, Nov 22, 2016 at 2:42 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> The first release candidate should be coming out this week. You can
> subscribe to the dev list if you want to follow the release schedule.
>
> On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com> wrote:
>
>> Hi Michael,
>>
>> I only see spark 2.0.2 which is what I am using currently. Any idea on
>> when 2.1 will be released?
>>
>> Thanks,
>> kant
>>
>> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> In Spark 2.1 we've added a from_json
>>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>>> function that I think will do what you want.
>>>
>>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com> wrote:
>>>
>>>> This seem to work
>>>>
>>>> import org.apache.spark.sql._
>>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>>> spark.read.json(rdd).show()
>>>>
>>>> However I wonder if this any inefficiency here ? since I have to apply
>>>> this function for billion rows.
>>>>
>>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by Michael Armbrust <mi...@databricks.com>.
The first release candidate should be coming out this week. You can
subscribe to the dev list if you want to follow the release schedule.
On Mon, Nov 21, 2016 at 9:34 PM, kant kodali <ka...@gmail.com> wrote:
> Hi Michael,
>
> I only see spark 2.0.2 which is what I am using currently. Any idea on
> when 2.1 will be released?
>
> Thanks,
> kant
>
> On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> In Spark 2.1 we've added a from_json
>> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
>> function that I think will do what you want.
>>
>> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com> wrote:
>>
>>> This seem to work
>>>
>>> import org.apache.spark.sql._
>>> val rdd = df2.rdd.map { case Row(j: String) => j }
>>> spark.read.json(rdd).show()
>>>
>>> However I wonder if this any inefficiency here ? since I have to apply
>>> this function for billion rows.
>>>
>>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by kant kodali <ka...@gmail.com>.
Hi Michael,
I only see spark 2.0.2 which is what I am using currently. Any idea on when
2.1 will be released?
Thanks,
kant
On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> In Spark 2.1 we've added a from_json
> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
> function that I think will do what you want.
>
> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com> wrote:
>
>> This seem to work
>>
>> import org.apache.spark.sql._
>> val rdd = df2.rdd.map { case Row(j: String) => j }
>> spark.read.json(rdd).show()
>>
>> However I wonder if this any inefficiency here ? since I have to apply
>> this function for billion rows.
>>
>>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by Michael Armbrust <mi...@databricks.com>.
In Spark 2.1 we've added a from_json
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
function that I think will do what you want.
On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <ka...@gmail.com> wrote:
> This seem to work
>
> import org.apache.spark.sql._
> val rdd = df2.rdd.map { case Row(j: String) => j }
> spark.read.json(rdd).show()
>
> However I wonder if this any inefficiency here ? since I have to apply
> this function for billion rows.
>
>
Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Posted by kant kodali <ka...@gmail.com>.
This seem to work
import org.apache.spark.sql._
val rdd = df2.rdd.map { case Row(j: String) => j }
spark.read.json(rdd).show()
However I wonder if this any inefficiency here ? since I have to apply this
function for billion rows.