You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/07 12:02:18 UTC
[GitHub] [spark] guiyanakuang commented on pull request #30467: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
guiyanakuang commented on pull request #30467:
URL: https://github.com/apache/spark/pull/30467#issuecomment-739875074
> @guiyanakuang, can we make it matched the examples you mentioned? Also, just to double check, we don't have any reference for the support nested array support, right?
Are you referring to this example ?
```sql
select c_column, d_column from table lateral view unnest(a.b.c, a.b.d) as c_column, d_column
```
Yes, we can implement this example, it's just that Unnest utdf does not yet exist for spark, and we can use explore instead.
```scala
val jsonStr1 = """{"a": [{"b": [{"c": [1,2]}]}]}"""
val jsonStr2 = """{"a": [{"b": [{"c": [1]}, {"c": [2]}]}]}"""
val df = spark.read.json(Seq(jsonStr1, jsonStr2).toDS())
df.createTempView("nested_arrays")
sql(
"""
|select
| c_u_u_u
|from
| nested_arrays
| lateral view explode(a.b.c) as c_u
| lateral view explode(c_u) as c_u_u
| lateral view explode(c_u_u) as c_u_u_u
|""".stripMargin).show()
+-------+
|c_u_u_u|
+-------+
| 1|
| 2|
| 1|
| 2|
+-------+
```
In fact, my company, Tencent, has implemented unnest in Spark, which is also referenced in Google's BigQuery.
Our advertising analysts rely heavily on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org