How to Register Permanent User-Defined-Functions (UDFs) in SparkSQL

[Lokesh Yadav <lo...@gmail.com>]

Hi
with sqlContext we can register a UDF like
this: sqlContext.udf.register("sample_fn", sample_fn _ )
But this UDF is limited to that particular sqlContext only. I wish to make
the registration persistent, so that I can access the same UDF in any
subsequent sqlcontext.
Or is there any other way to register UDFs in sparkSQL so that they remain
persistent?

Regards
Lokesh

Re: How to Register Permanent User-Defined-Functions (UDFs) in SparkSQL

[Daniel Darabos <da...@lynxanalytics.com>]

Hi Lokesh,
There is no way to do that. SqlContext.newSession documentation says:

    Returns a SQLContext as new session, with separated SQL configurations,
temporary tables, registered functions, but sharing the same SparkContext,
CacheManager, SQLListener and SQLTab.

You have two options: either use the same SQLContext instead of creating
new SQLContexts, or have a function for creating SQLContexts, and this
function can also register the UDFs in every created SQLContext.

On Sun, Jul 10, 2016 at 6:14 PM, Lokesh Yadav <lo...@gmail.com>
wrote:

> Hi
> with sqlContext we can register a UDF like
> this: sqlContext.udf.register("sample_fn", sample_fn _ )
> But this UDF is limited to that particular sqlContext only. I wish to make
> the registration persistent, so that I can access the same UDF in any
> subsequent sqlcontext.
> Or is there any other way to register UDFs in sparkSQL so that they remain
> persistent?
>
> Regards
> Lokesh
>