You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jason White <ja...@shopify.com> on 2016/09/25 22:13:30 UTC
ArrayType support in Spark SQL
It seems that `functions.lit` doesn't support ArrayTypes. To reproduce:
org.apache.spark.sql.functions.lit(2 :: 1 :: Nil)
java.lang.RuntimeException: Unsupported literal type class
scala.collection.immutable.$colon$colon List(2, 1)
at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:59)
at org.apache.spark.sql.functions$.lit(functions.scala:101)
... 48 elided
This is about the first thing I tried to do with ArrayTypes in Spark SQL. Is
this usage supported, or on the roadmap?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/ArrayType-support-in-Spark-SQL-tp19063.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: ArrayType support in Spark SQL
Posted by Koert Kuipers <ko...@tresata.com>.
not pretty but this works:
import org.apache.spark.sql.functions.udf
df.withColumn("array", sqlf.udf({ () => Seq(1, 2, 3) }).apply())
On Sun, Sep 25, 2016 at 6:13 PM, Jason White <ja...@shopify.com>
wrote:
> It seems that `functions.lit` doesn't support ArrayTypes. To reproduce:
>
> org.apache.spark.sql.functions.lit(2 :: 1 :: Nil)
>
> java.lang.RuntimeException: Unsupported literal type class
> scala.collection.immutable.$colon$colon List(2, 1)
> at
> org.apache.spark.sql.catalyst.expressions.Literal$.apply(
> literals.scala:59)
> at org.apache.spark.sql.functions$.lit(functions.scala:101)
> ... 48 elided
>
> This is about the first thing I tried to do with ArrayTypes in Spark SQL.
> Is
> this usage supported, or on the roadmap?
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/ArrayType-support-
> in-Spark-SQL-tp19063.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: ArrayType support in Spark SQL
Posted by Takeshi Yamamuro <li...@gmail.com>.
Done: https://issues.apache.org/jira/browse/SPARK-17683
On Tue, Sep 27, 2016 at 5:43 AM, Reynold Xin <rx...@databricks.com> wrote:
> Seems fair & easy to support. Can somebody open a JIRA ticket and patch?
>
>
> On Mon, Sep 26, 2016 at 9:05 AM, Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Since `Literal#default` can handle array types, it seems there is no
>> strong reason
>> for unsupporting the type in `Literal#apply`, that is, `functions.lit`.
>> https://github.com/apache/spark/blob/master/sql/catalyst/
>> src/main/scala/org/apache/spark/sql/catalyst/expressions
>> /literals.scala#L119
>>
>> // maropu
>>
>> On Mon, Sep 26, 2016 at 7:58 AM, Jason White <ja...@shopify.com>
>> wrote:
>>
>>> Continuing to dig, I encountered:
>>> https://github.com/apache/spark/blob/master/sql/catalyst/src
>>> /test/scala/org/apache/spark/sql/catalyst/expressions/
>>> LiteralExpressionSuite.scala#L125
>>>
>>> // TODO(davies): add tests for ArrayType, MapType and StructType
>>>
>>> I guess others have thought of this already, just not implemented yet. :)
>>>
>>> For others reading this thread, someone suggested using a SQL UDF to
>>> return
>>> the constant - this works as a hack for now.
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-spark-developers
>>> -list.1001551.n3.nabble.com/ArrayType-support-in-Spark-SQL-
>>> tp19063p19065.html
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>
>
--
---
Takeshi Yamamuro
Re: ArrayType support in Spark SQL
Posted by Reynold Xin <rx...@databricks.com>.
Seems fair & easy to support. Can somebody open a JIRA ticket and patch?
On Mon, Sep 26, 2016 at 9:05 AM, Takeshi Yamamuro <li...@gmail.com>
wrote:
> Hi,
>
> Since `Literal#default` can handle array types, it seems there is no
> strong reason
> for unsupporting the type in `Literal#apply`, that is, `functions.lit`.
> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/literals.scala#L119
>
> // maropu
>
> On Mon, Sep 26, 2016 at 7:58 AM, Jason White <ja...@shopify.com>
> wrote:
>
>> Continuing to dig, I encountered:
>> https://github.com/apache/spark/blob/master/sql/catalyst/
>> src/test/scala/org/apache/spark/sql/catalyst/expressions
>> /LiteralExpressionSuite.scala#L125
>>
>> // TODO(davies): add tests for ArrayType, MapType and StructType
>>
>> I guess others have thought of this already, just not implemented yet. :)
>>
>> For others reading this thread, someone suggested using a SQL UDF to
>> return
>> the constant - this works as a hack for now.
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-developers
>> -list.1001551.n3.nabble.com/ArrayType-support-in-Spark-
>> SQL-tp19063p19065.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> ---
> Takeshi Yamamuro
>
Re: ArrayType support in Spark SQL
Posted by Takeshi Yamamuro <li...@gmail.com>.
Hi,
Since `Literal#default` can handle array types, it seems there is no strong
reason
for unsupporting the type in `Literal#apply`, that is, `functions.lit`.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala#L119
// maropu
On Mon, Sep 26, 2016 at 7:58 AM, Jason White <ja...@shopify.com>
wrote:
> Continuing to dig, I encountered:
> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/
> LiteralExpressionSuite.scala#L125
>
> // TODO(davies): add tests for ArrayType, MapType and StructType
>
> I guess others have thought of this already, just not implemented yet. :)
>
> For others reading this thread, someone suggested using a SQL UDF to return
> the constant - this works as a hack for now.
>
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/ArrayType-support-
> in-Spark-SQL-tp19063p19065.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
---
Takeshi Yamamuro
Re: ArrayType support in Spark SQL
Posted by Jason White <ja...@shopify.com>.
Continuing to dig, I encountered:
https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala#L125
// TODO(davies): add tests for ArrayType, MapType and StructType
I guess others have thought of this already, just not implemented yet. :)
For others reading this thread, someone suggested using a SQL UDF to return
the constant - this works as a hack for now.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/ArrayType-support-in-Spark-SQL-tp19063p19065.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org