You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Erik Erlandson <ee...@redhat.com> on 2017/07/06 00:33:28 UTC

UDAFs for sketching Dataset columns with T-Digests

After my talk on T-Digests in Spark at Spark Summit East, there were some
requests for a UDAF-based interface for working with Datasets.   I'm
pleased to announce that I released a library for doing T-Digest sketching
with UDAFs:

https://github.com/isarn/isarn-sketches-spark

This initial release provides support for Scala. Future releases will
support PySpark bindings, and additional tools for leveraging T-Digests in
ML pipelines.

Cheers!
Erik

Re: UDAFs for sketching Dataset columns with T-Digests

Posted by Sam Bessalah <sa...@gmail.com>.

This is interesting and very useful.
Thanks.

On Thu, Jul 6, 2017 at 2:33 AM, Erik Erlandson <ee...@redhat.com> wrote:

> After my talk on T-Digests in Spark at Spark Summit East, there were some
> requests for a UDAF-based interface for working with Datasets.   I'm
> pleased to announce that I released a library for doing T-Digest sketching
> with UDAFs:
>
> https://github.com/isarn/isarn-sketches-spark
>
> This initial release provides support for Scala. Future releases will
> support PySpark bindings, and additional tools for leveraging T-Digests in
> ML pipelines.
>
> Cheers!
> Erik
>