You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jakub Dubovsky <sp...@gmail.com> on 2016/12/15 16:18:47 UTC
Dataset encoders for further types?
Hey,
I want to ask whether there is any roadmap/plan for adding Encoders for
further types in next releases of Spark. Here is a list
<http://spark.apache.org/docs/latest/sql-programming-guide.html#data-types> of
currently supported types. We would like to use Datasets with our
internally defined case classes containing
scala.collection.immutable.List(s). This does not work now because these
lists are converted to ArrayType (Seq). This then fails a constructor
lookup because of seq-is-not-a-list error...
This means that for now we are stuck with using RDDs.
Thanks for any insights!
Jakub Dubovsky
Re: Dataset encoders for further types?
Posted by Michal Šenkýř <mi...@gmail.com>.
I actually already made a pull request adding support for arbitrary
sequence types.
https://github.com/apache/spark/pull/16240
There is still a little problem of Seq.toDS not working for those types
(couldn't get implicits with multiple type parameters to resolve
correctly) but createDataset works fine.
Would be glad if you bring some attention to it. It's my first
code-related pull request and noone responded to it yet. I'm wondering
if I'm doing something wrong on that front.
Michal Senkyr
On 16.12.2016 12:04, Jakub Dubovsky wrote:
> I will give that a try. Thanks!
>
> On Fri, Dec 16, 2016 at 12:45 AM, Michael Armbrust
> <michael@databricks.com <ma...@databricks.com>> wrote:
>
> I would have sworn there was a ticket, but I can't find it. So
> here you go: https://issues.apache.org/jira/browse/SPARK-18891
> <https://issues.apache.org/jira/browse/SPARK-18891>
>
> A work around until that is fixed would be for you to manually
> specify the kryo encoder
> <http://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/sql/Encoders.html#kryo%28scala.reflect.ClassTag%29>.
>
> On Thu, Dec 15, 2016 at 8:18 AM, Jakub Dubovsky
> <spark.dubovsky.jakub@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hey,
>
> I want to ask whether there is any roadmap/plan for adding
> Encoders for further types in next releases of Spark. Here is
> a list
> <http://spark.apache.org/docs/latest/sql-programming-guide.html#data-types> of
> currently supported types. We would like to use Datasets with
> our internally defined case classes containing
> scala.collection.immutable.List(s). This does not work now
> because these lists are converted to ArrayType (Seq). This
> then fails a constructor lookup because of seq-is-not-a-list
> error...
>
> This means that for now we are stuck with using RDDs.
>
> Thanks for any insights!
>
> Jakub Dubovsky
>
>
>
Re: Dataset encoders for further types?
Posted by Jakub Dubovsky <sp...@gmail.com>.
I will give that a try. Thanks!
On Fri, Dec 16, 2016 at 12:45 AM, Michael Armbrust <mi...@databricks.com>
wrote:
> I would have sworn there was a ticket, but I can't find it. So here you
> go: https://issues.apache.org/jira/browse/SPARK-18891
>
> A work around until that is fixed would be for you to manually specify the kryo
> encoder
> <http://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/sql/Encoders.html#kryo(scala.reflect.ClassTag)>
> .
>
> On Thu, Dec 15, 2016 at 8:18 AM, Jakub Dubovsky <
> spark.dubovsky.jakub@gmail.com> wrote:
>
>> Hey,
>>
>> I want to ask whether there is any roadmap/plan for adding Encoders for
>> further types in next releases of Spark. Here is a list
>> <http://spark.apache.org/docs/latest/sql-programming-guide.html#data-types> of
>> currently supported types. We would like to use Datasets with our
>> internally defined case classes containing scala.collection.immutable.List(s).
>> This does not work now because these lists are converted to ArrayType
>> (Seq). This then fails a constructor lookup because of seq-is-not-a-list
>> error...
>>
>> This means that for now we are stuck with using RDDs.
>>
>> Thanks for any insights!
>>
>> Jakub Dubovsky
>>
>>
>
Re: Dataset encoders for further types?
Posted by Michael Armbrust <mi...@databricks.com>.
I would have sworn there was a ticket, but I can't find it. So here you
go: https://issues.apache.org/jira/browse/SPARK-18891
A work around until that is fixed would be for you to manually specify the kryo
encoder
<http://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/sql/Encoders.html#kryo(scala.reflect.ClassTag)>
.
On Thu, Dec 15, 2016 at 8:18 AM, Jakub Dubovsky <
spark.dubovsky.jakub@gmail.com> wrote:
> Hey,
>
> I want to ask whether there is any roadmap/plan for adding Encoders for
> further types in next releases of Spark. Here is a list
> <http://spark.apache.org/docs/latest/sql-programming-guide.html#data-types> of
> currently supported types. We would like to use Datasets with our
> internally defined case classes containing scala.collection.immutable.List(s).
> This does not work now because these lists are converted to ArrayType
> (Seq). This then fails a constructor lookup because of seq-is-not-a-list
> error...
>
> This means that for now we are stuck with using RDDs.
>
> Thanks for any insights!
>
> Jakub Dubovsky
>
>