You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by vinod kumar <vi...@gmail.com> on 2015/07/08 10:53:15 UTC

UDF in spark

Hi Everyone,

I am new to spark.may I know how to define and use User Define Function in
SPARK SQL.

I want to use defined UDF by using sql queries.

My Environment
Windows 8
spark 1.3.1

Warm Regards,
Vinod

Re: UDF in spark

Posted by vinod kumar <vi...@gmail.com>.

Thank you for quick response Vishnu,

I have following doubts too.

1.Is there is anyway to upload files to HDFS programattically using c#
language?.
2.Is there is any way to automatically load scala block of code (for UDF)
when i start the spark service?

-Vinod

On Wed, Jul 8, 2015 at 7:57 AM, VISHNU SUBRAMANIAN <
johnfedrickenator@gmail.com> wrote:

> HI Vinod,
>
> Yes If you want to use a scala or python function you need the block of
> code.
>
> Only Hive UDF's are available permanently.
>
> Thanks,
> Vishnu
>
> On Wed, Jul 8, 2015 at 5:17 PM, vinod kumar <vi...@gmail.com>
> wrote:
>
>> Thanks Vishnu,
>>
>> When restart the service the UDF was not accessible by my query.I need to
>> run the mentioned block again to use the UDF.
>> Is there is any way to maintain UDF in sqlContext permanently?
>>
>> Thanks,
>> Vinod
>>
>> On Wed, Jul 8, 2015 at 7:16 AM, VISHNU SUBRAMANIAN <
>> johnfedrickenator@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> sqlContext.udf.register("udfname", functionname _)
>>>
>>> example:
>>>
>>> def square(x:Int):Int = { x * x}
>>>
>>> register udf as below
>>>
>>> sqlContext.udf.register("square",square _)
>>>
>>> Thanks,
>>> Vishnu
>>>
>>> On Wed, Jul 8, 2015 at 2:23 PM, vinod kumar <vi...@gmail.com>
>>> wrote:
>>>
>>>> Hi Everyone,
>>>>
>>>> I am new to spark.may I know how to define and use User Define Function
>>>> in SPARK SQL.
>>>>
>>>> I want to use defined UDF by using sql queries.
>>>>
>>>> My Environment
>>>> Windows 8
>>>> spark 1.3.1
>>>>
>>>> Warm Regards,
>>>> Vinod
>>>>
>>>>
>>>>
>>>
>>
>

Re: UDF in spark

Posted by VISHNU SUBRAMANIAN <jo...@gmail.com>.

HI Vinod,

Yes If you want to use a scala or python function you need the block of
code.

Only Hive UDF's are available permanently.

Thanks,
Vishnu

On Wed, Jul 8, 2015 at 5:17 PM, vinod kumar <vi...@gmail.com>
wrote:

> Thanks Vishnu,
>
> When restart the service the UDF was not accessible by my query.I need to
> run the mentioned block again to use the UDF.
> Is there is any way to maintain UDF in sqlContext permanently?
>
> Thanks,
> Vinod
>
> On Wed, Jul 8, 2015 at 7:16 AM, VISHNU SUBRAMANIAN <
> johnfedrickenator@gmail.com> wrote:
>
>> Hi,
>>
>> sqlContext.udf.register("udfname", functionname _)
>>
>> example:
>>
>> def square(x:Int):Int = { x * x}
>>
>> register udf as below
>>
>> sqlContext.udf.register("square",square _)
>>
>> Thanks,
>> Vishnu
>>
>> On Wed, Jul 8, 2015 at 2:23 PM, vinod kumar <vi...@gmail.com>
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I am new to spark.may I know how to define and use User Define Function
>>> in SPARK SQL.
>>>
>>> I want to use defined UDF by using sql queries.
>>>
>>> My Environment
>>> Windows 8
>>> spark 1.3.1
>>>
>>> Warm Regards,
>>> Vinod
>>>
>>>
>>>
>>
>

Re: UDF in spark

Posted by vinod kumar <vi...@gmail.com>.

Thanks Vishnu,

When restart the service the UDF was not accessible by my query.I need to
run the mentioned block again to use the UDF.
Is there is any way to maintain UDF in sqlContext permanently?

Thanks,
Vinod

On Wed, Jul 8, 2015 at 7:16 AM, VISHNU SUBRAMANIAN <
johnfedrickenator@gmail.com> wrote:

> Hi,
>
> sqlContext.udf.register("udfname", functionname _)
>
> example:
>
> def square(x:Int):Int = { x * x}
>
> register udf as below
>
> sqlContext.udf.register("square",square _)
>
> Thanks,
> Vishnu
>
> On Wed, Jul 8, 2015 at 2:23 PM, vinod kumar <vi...@gmail.com>
> wrote:
>
>> Hi Everyone,
>>
>> I am new to spark.may I know how to define and use User Define Function
>> in SPARK SQL.
>>
>> I want to use defined UDF by using sql queries.
>>
>> My Environment
>> Windows 8
>> spark 1.3.1
>>
>> Warm Regards,
>> Vinod
>>
>>
>>
>

Re: UDF in spark

Posted by VISHNU SUBRAMANIAN <jo...@gmail.com>.

Hi,

sqlContext.udf.register("udfname", functionname _)

example:

def square(x:Int):Int = { x * x}

register udf as below

sqlContext.udf.register("square",square _)

Thanks,
Vishnu

On Wed, Jul 8, 2015 at 2:23 PM, vinod kumar <vi...@gmail.com>
wrote:

> Hi Everyone,
>
> I am new to spark.may I know how to define and use User Define Function in
> SPARK SQL.
>
> I want to use defined UDF by using sql queries.
>
> My Environment
> Windows 8
> spark 1.3.1
>
> Warm Regards,
> Vinod
>
>
>