You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anyweil <we...@gmail.com> on 2014/07/13 13:11:56 UTC
Re: Nested Query With Spark SQL(1.0.1)
Or is it supported? I know I could doing it myself with filter, but if SQL
could support, would be much better, thx!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Query-With-Spark-SQL-1-0-1-tp9544p9547.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Nested Query With Spark SQL(1.0.1)
Posted by anyweil <we...@gmail.com>.
Yes, just as my last post, using [] to access array data and "." to access
nested fields seems not work.
BTW, i have deeped into the code of the current master branch.
spark / sql / catalyst / src / main / scala / org / apache / spark / sql /
catalyst / plans / logical / LogicalPlan.scala
from line 72 (in the function def resolve(name: String):
Option[NamedExpression])
...
options.distinct match {
case (a, Nil) :: Nil => Some(a) // One match, no nested fields, use
it.
// One match, but we also need to extract the requested nested field.
case (a, nestedFields) :: Nil =>
a.dataType match {
case StructType(fields) =>
Some(Alias(nestedFields.foldLeft(a: Expression)(GetField),
nestedFields.last)())
case _ => None // Don't know how to resolve these field references
}
case Nil => None // No matches.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name:
${ambiguousReferences.mkString(",")}")
}
...
seems only StructType is handled and the ArrayType is not.
So it seems to be bug or some feature not finished yet?
thx!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-the-nested-JSON-data-With-Spark-SQL-1-0-1-tp9544p9733.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Nested Query With Spark SQL(1.0.1)
Posted by Michael Armbrust <mi...@databricks.com>.
In general this should be supported using [] to access array data and "."
to access nested fields. Is there something you are trying that isn't
working?
On Mon, Jul 14, 2014 at 11:25 PM, anyweil <we...@gmail.com> wrote:
> I mean the query on the nested data such as JSON, not the nested query,
> sorry
> for the misunderstanding.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Query-the-nested-JSON-data-With-Spark-SQL-1-0-1-tp9544p9726.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Nested Query With Spark SQL(1.0.1)
Posted by anyweil <we...@gmail.com>.
I mean the query on the nested data such as JSON, not the nested query, sorry
for the misunderstanding.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-the-nested-JSON-data-With-Spark-SQL-1-0-1-tp9544p9726.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Nested Query With Spark SQL(1.0.1)
Posted by Michael Armbrust <mi...@databricks.com>.
What sort of nested query are you talking about? Right now we only support
nested queries in the FROM clause. I'd like to add support for other cases
in the future.
On Sun, Jul 13, 2014 at 4:11 AM, anyweil <we...@gmail.com> wrote:
> Or is it supported? I know I could doing it myself with filter, but if SQL
> could support, would be much better, thx!
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Query-With-Spark-SQL-1-0-1-tp9544p9547.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>