You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by raghukiran <ra...@gmail.com> on 2016/01/18 18:57:50 UTC
Spark SQL create table
Is creating a table using the SparkSQLContext currently supported?
Regards,
Raghu
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark SQL create table
Posted by Ted Yu <yu...@gmail.com>.
Have you taken a look
at sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala ?
You can find examples there.
On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com> wrote:
> Is creating a table using the SparkSQLContext currently supported?
>
> Regards,
> Raghu
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark SQL create table
Posted by Raghu Ganti <ra...@gmail.com>.
Great, I got that to work following your example! Thanks.
A followup question is: If I had a custom SQL type (UserDefinedType<T>),
how can I map it to this type from the RDD in the DataFrame?
Regards
On Mon, Jan 18, 2016 at 1:35 PM, Ted Yu <yu...@gmail.com> wrote:
> By SparkSQLContext, I assume you mean SQLContext.
> From the doc for SQLContext#createDataFrame():
>
> * dataFrame.registerTempTable("people")
> * sqlContext.sql("select name from people").collect.foreach(println)
>
> If you want to persist table externally, you need Hive, etc
>
> Regards
>
> On Mon, Jan 18, 2016 at 10:28 AM, Raghu Ganti <ra...@gmail.com>
> wrote:
>
>> This requires Hive to be installed and uses HiveContext, right?
>>
>> What is the SparkSQLContext useful for?
>>
>> On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Please take a look
>>> at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
>>>
>>> On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com>
>>> wrote:
>>>
>>>> Is creating a table using the SparkSQLContext currently supported?
>>>>
>>>> Regards,
>>>> Raghu
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: Spark SQL create table
Posted by Ted Yu <yu...@gmail.com>.
By SparkSQLContext, I assume you mean SQLContext.
>From the doc for SQLContext#createDataFrame():
* dataFrame.registerTempTable("people")
* sqlContext.sql("select name from people").collect.foreach(println)
If you want to persist table externally, you need Hive, etc
Regards
On Mon, Jan 18, 2016 at 10:28 AM, Raghu Ganti <ra...@gmail.com> wrote:
> This requires Hive to be installed and uses HiveContext, right?
>
> What is the SparkSQLContext useful for?
>
> On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please take a look
>> at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
>>
>> On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com> wrote:
>>
>>> Is creating a table using the SparkSQLContext currently supported?
>>>
>>> Regards,
>>> Raghu
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Spark SQL create table
Posted by Raghu Ganti <ra...@gmail.com>.
Btw, Thanks a lot for all your quick responses - it is very useful and
definitely appreciate it :-)
On Mon, Jan 18, 2016 at 1:28 PM, Raghu Ganti <ra...@gmail.com> wrote:
> This requires Hive to be installed and uses HiveContext, right?
>
> What is the SparkSQLContext useful for?
>
> On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please take a look
>> at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
>>
>> On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com> wrote:
>>
>>> Is creating a table using the SparkSQLContext currently supported?
>>>
>>> Regards,
>>> Raghu
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Spark SQL create table
Posted by Raghu Ganti <ra...@gmail.com>.
This requires Hive to be installed and uses HiveContext, right?
What is the SparkSQLContext useful for?
On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look
> at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
>
> On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com> wrote:
>
>> Is creating a table using the SparkSQLContext currently supported?
>>
>> Regards,
>> Raghu
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark SQL create table
Posted by Ted Yu <yu...@gmail.com>.
Please take a look
at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
On Mon, Jan 18, 2016 at 9:57 AM, raghukiran <ra...@gmail.com> wrote:
> Is creating a table using the SparkSQLContext currently supported?
>
> Regards,
> Raghu
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-create-table-tp25996.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>