Re: Catalyst - ObjectType for Encoders

[Aleksander Eskilson <al...@gmail.com>]

Sounds good to me, I'll keep that in mind and pay attention to updates.
Shall I make a pull request from my fork, or would someone else prefer to
change those function signatures themselves?

Thanks,
Alek

On Fri, Sep 30, 2016 at 2:25 PM Michael Armbrust <mi...@databricks.com>
wrote:

> I'd be okay removing that modifier, with one caveat.  The code in
> org.apache.spark.sql.catalyst.* is purposefully excluded from published
> documentation and does not have the same compatibility guarantees as the
> rest of the Spark's Public APIs.  We leave most of it not "private" so that
> advanced users can experiment.
>
> So, you can write a custom encoder, but until we finalize the encoder API,
> you might need to update it each Spark release.
>
> On Fri, Sep 30, 2016 at 6:36 AM, Aleksander Eskilson <
> aleksanderesk@gmail.com> wrote:
>
> Hi there,
>
>
>
> Currently Catalyst supports encoding custom classes represented as Java
> Beans (among others). This Java Bean implementation depends internally on
> Catalyst’s ObjectType extension of DataType. Currently, this class is
> private to the sql package [1], which is sensible, as it is only necessary
> for representing objects in the encoder. However, its private scope makes
> it more difficult (if not impossible) to write full custom encoders for
> other classes, themselves perhaps composed of additional objects.
>
>
>
> Could the definition of the ObjectType be made public in order to support
> writing custom Encoders? I can't see any particular danger in this.
>
> In order to implement an encoder for Avro-specified classes, I have to
> work off a fork where the ObjectType definition has be liberalized as
> described.
>
>
>
> [1] --
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala#L39
>
>
>
> Thanks,
>
> Aleksander Eskilson
>
>
>