You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "vdiwakar.malladi" <vd...@gmail.com> on 2014/09/29 17:08:28 UTC
Unresolved attributes: SparkSQL on the schemaRDD
Hello,
I'm exploring SparkSQL and I'm facing issue while using the queries. Any
help on this is appreciated.
I have the following schema once loaded as RDD.
root
|-- data: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- age: integer (nullable = true)
| | |-- name: string (nullable = true)
my people.json is as follows.
{"data": [{"name":"Michael"}, {"name":"Andy", "age":30}, {"name":"Justin",
"age":19}]}
when I'm querying "select name from people where age>=13"; I'm getting the
following exception.
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
attributes: 'element.name, tree:
Project ['name]
Filter ('age >= 13)
Subquery people
My question is how to access name, age from root node to query the same.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by Yin Huai <hu...@gmail.com>.
I think this problem has been fixed after the 1.1 release. Can you try the
master branch?
On Mon, Sep 29, 2014 at 10:06 PM, vdiwakar.malladi <
vdiwakar.malladi@gmail.com> wrote:
> I'm using the latest version i.e. Spark 1.1.0
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15376.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
I'm using the latest version i.e. Spark 1.1.0
Thanks.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15376.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Sorry. Not that, this one
http://arjon.es/2014/07/01/processing-json-with-spark-sql/
Thanks
Best Regards
On Tue, Sep 30, 2014 at 1:43 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> This one explains it nicely
> http://www.dofactory.com/topic/1816/spark-performing-a-join-and-getting-results-back-in-a-strongly-typed-collection.aspx
>
> Thanks
> Best Regards
>
> On Tue, Sep 30, 2014 at 12:57 AM, Yin Huai <hu...@gmail.com> wrote:
>
>> What version of Spark did you use? Can you try the master branch?
>>
>> On Mon, Sep 29, 2014 at 1:52 PM, vdiwakar.malladi <
>> vdiwakar.malladi@gmail.com> wrote:
>>
>>> Thanks for your prompt response.
>>>
>>> Still on further note, I'm getting the exception while executing the
>>> query.
>>>
>>> "SELECT data[0].name FROM people where data[0].age >=13"
>>>
>>> *Exception in thread "main" java.lang.RuntimeException: [1.46] failure:
>>> ``UNION'' expected but identifier .age found
>>>
>>> SELECT data[0].name FROM people where data[0].age >=13*
>>>
>>>
>>> Am I missing something? I'm trying to understand how the RDD is stored.
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15350.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
This one explains it nicely
http://www.dofactory.com/topic/1816/spark-performing-a-join-and-getting-results-back-in-a-strongly-typed-collection.aspx
Thanks
Best Regards
On Tue, Sep 30, 2014 at 12:57 AM, Yin Huai <hu...@gmail.com> wrote:
> What version of Spark did you use? Can you try the master branch?
>
> On Mon, Sep 29, 2014 at 1:52 PM, vdiwakar.malladi <
> vdiwakar.malladi@gmail.com> wrote:
>
>> Thanks for your prompt response.
>>
>> Still on further note, I'm getting the exception while executing the
>> query.
>>
>> "SELECT data[0].name FROM people where data[0].age >=13"
>>
>> *Exception in thread "main" java.lang.RuntimeException: [1.46] failure:
>> ``UNION'' expected but identifier .age found
>>
>> SELECT data[0].name FROM people where data[0].age >=13*
>>
>>
>> Am I missing something? I'm trying to understand how the RDD is stored.
>>
>> Thanks in advance.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15350.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by Yin Huai <hu...@gmail.com>.
What version of Spark did you use? Can you try the master branch?
On Mon, Sep 29, 2014 at 1:52 PM, vdiwakar.malladi <
vdiwakar.malladi@gmail.com> wrote:
> Thanks for your prompt response.
>
> Still on further note, I'm getting the exception while executing the query.
>
> "SELECT data[0].name FROM people where data[0].age >=13"
>
> *Exception in thread "main" java.lang.RuntimeException: [1.46] failure:
> ``UNION'' expected but identifier .age found
>
> SELECT data[0].name FROM people where data[0].age >=13*
>
>
> Am I missing something? I'm trying to understand how the RDD is stored.
>
> Thanks in advance.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15350.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
Thanks for your prompt response.
Still on further note, I'm getting the exception while executing the query.
"SELECT data[0].name FROM people where data[0].age >=13"
*Exception in thread "main" java.lang.RuntimeException: [1.46] failure:
``UNION'' expected but identifier .age found
SELECT data[0].name FROM people where data[0].age >=13*
Am I missing something? I'm trying to understand how the RDD is stored.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15350.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unresolved attributes: SparkSQL on the schemaRDD
Posted by Cheng Lian <li...@gmail.com>.
In your case, the table has only one row, whose contents is “data”,
which is an array. You need something like |SELECT data[0].name FROM
json_table| to access the |name| field.
On 9/29/14 11:08 PM, vdiwakar.malladi wrote:
> Hello,
>
> I'm exploring SparkSQL and I'm facing issue while using the queries. Any
> help on this is appreciated.
>
> I have the following schema once loaded as RDD.
>
> root
> |-- data: array (nullable = true)
> | |-- element: struct (containsNull = false)
> | | |-- age: integer (nullable = true)
> | | |-- name: string (nullable = true)
>
>
> my people.json is as follows.
>
> {"data": [{"name":"Michael"}, {"name":"Andy", "age":30}, {"name":"Justin",
> "age":19}]}
>
> when I'm querying "select name from people where age>=13"; I'm getting the
> following exception.
>
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
> attributes: 'element.name, tree:
> Project ['name]
> Filter ('age >= 13)
> Subquery people
>
> My question is how to access name, age from root node to query the same.
>
> Thanks in advance.
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>