You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Divya Gehlot <di...@gmail.com> on 2016/07/20 07:14:47 UTC
write and call UDF in spark dataframe
Hi,
Could somebody share example of writing and calling udf which converts unix
tme stamp to date tiime .
Thanks,
Divya
Re: write and call UDF in spark dataframe
Posted by Jacek Laskowski <ja...@japila.pl>.
On Wed, Jul 20, 2016 at 1:22 PM, Rishabh Bhardwaj <rb...@gmail.com> wrote:
> val new_df = df.select(from_unixtime($"time").as("newtime"))
or better yet using tick (less typing and more prose than code :))
df.select(from_unixtime('time) as "newtime")
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: write and call UDF in spark dataframe
Posted by Mich Talebzadeh <mi...@gmail.com>.
yep something in line of
val df = sqlContext.sql("SELECT FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy
HH:mm:ss.ss') as time ")
Note that this does not require a column from an already existing table.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 20 July 2016 at 12:22, Rishabh Bhardwaj <rb...@gmail.com> wrote:
> Hi Divya,
>
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in
> dataframe DSL you can try like this,
>
> val new_df = df.select(from_unixtime($"time").as("newtime"))
>
>
> Thanks,
> Rishabh.
>
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
> dev.rabin.banerjee@gmail.com> wrote:
>
>> Hi Divya ,
>>
>> Try,
>>
>> val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr")
>>
>> Regards,
>> Rabin
>>
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts
>>> unix tme stamp to date tiime .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
Re: write and call UDF in spark dataframe
Posted by Kabeer Ahmed <ka...@outlook.com>.
Divya:
https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html
The link gives a complete example of registering a udAf - user defined aggregate function. This is a complete example and this example should give you a complete idea of registering a UDF. If you still need a hand let us know.
Thanks
Kabeer.
Sent from Nylas N1<https://link.nylas.com/link/6fso0l6fy8gw777bnnxgxerwm/local-34e17349-235c/0?redirect=https%3A%2F%2Fnylas.com%2Fn1%3Fref%3Dn1>, the extensible, open source mail client.
On Jul 21 2016, at 8:13 am, Jacek Laskowski <ja...@japila.pl> wrote:
On Thu, Jul 21, 2016 at 5:53 AM, Mich Talebzadeh
<mi...@gmail.com> wrote:
> something similar
Is this going to be in Scala?
> def ChangeToDate (word : String) : Date = {
> //return
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/yyyy"),"yyyy-MM-dd"))
> val d1 = Date.valueOf(ReverseDate(word))
> return d1
> }
> sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))
then...please use lowercase method names and *no* return please ;-)
BTW, no sqlContext as of Spark 2.0. Sorry.../me smiling nicely
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: write and call UDF in spark dataframe
Posted by Jacek Laskowski <ja...@japila.pl>.
On Thu, Jul 21, 2016 at 5:53 AM, Mich Talebzadeh
<mi...@gmail.com> wrote:
> something similar
Is this going to be in Scala?
> def ChangeToDate (word : String) : Date = {
> //return
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/yyyy"),"yyyy-MM-dd"))
> val d1 = Date.valueOf(ReverseDate(word))
> return d1
> }
> sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))
then...please use lowercase method names and *no* return please ;-)
BTW, no sqlContext as of Spark 2.0. Sorry.../me smiling nicely
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: write and call UDF in spark dataframe
Posted by Mich Talebzadeh <mi...@gmail.com>.
something similar
def ChangeToDate (word : String) : Date = {
//return
TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/yyyy"),"yyyy-MM-dd"))
val d1 = Date.valueOf(ReverseDate(word))
return d1
}
sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 21 July 2016 at 03:53, Divya Gehlot <di...@gmail.com> wrote:
> Hi ,
> To be very specific I am looking for UDFs syntax for example which takes
> String as parameter and returns integer .. how do we define the return type
> .
>
>
> Thanks,
>
> Divya
>
> On 21 July 2016 at 00:24, Andy Davidson <An...@santacruzintegration.com>
> wrote:
>
>> Hi Divya
>>
>> In general you will get better performance if you can minimize your use
>> of UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
>> treat your UDF as a block box.
>>
>> Andy
>>
>> From: Rishabh Bhardwaj <rb...@gmail.com>
>> Date: Wednesday, July 20, 2016 at 4:22 AM
>> To: Rabin Banerjee <de...@gmail.com>
>> Cc: Divya Gehlot <di...@gmail.com>, "user @spark" <
>> user@spark.apache.org>
>> Subject: Re: write and call UDF in spark dataframe
>>
>> Hi Divya,
>>
>> There is already "from_unixtime" exists in
>> org.apache.spark.sql.frunctions,
>> Rabin has used that in the sql query,if you want to use it in
>> dataframe DSL you can try like this,
>>
>> val new_df = df.select(from_unixtime($"time").as("newtime"))
>>
>>
>> Thanks,
>> Rishabh.
>>
>> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
>> dev.rabin.banerjee@gmail.com> wrote:
>>
>>> Hi Divya ,
>>>
>>> Try,
>>>
>>> val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr")
>>>
>>> Regards,
>>> Rabin
>>>
>>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Could somebody share example of writing and calling udf which converts
>>>> unix tme stamp to date tiime .
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>
>>>
>>
>
Re: write and call UDF in spark dataframe
Posted by Jacek Laskowski <ja...@japila.pl>.
On Thu, Jul 21, 2016 at 4:53 AM, Divya Gehlot <di...@gmail.com> wrote:
> To be very specific I am looking for UDFs syntax for example which takes
> String as parameter and returns integer .. how do we define the return type
val f: String => Int = ???
val myUDF = udf(f)
or
val myUDF = udf[String, Int] { ??? }
or
val myUDF = udf { (s: String) => ??? }
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: write and call UDF in spark dataframe
Posted by Divya Gehlot <di...@gmail.com>.
Hi ,
To be very specific I am looking for UDFs syntax for example which takes
String as parameter and returns integer .. how do we define the return type
.
Thanks,
Divya
On 21 July 2016 at 00:24, Andy Davidson <An...@santacruzintegration.com>
wrote:
> Hi Divya
>
> In general you will get better performance if you can minimize your use of
> UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
> treat your UDF as a block box.
>
> Andy
>
> From: Rishabh Bhardwaj <rb...@gmail.com>
> Date: Wednesday, July 20, 2016 at 4:22 AM
> To: Rabin Banerjee <de...@gmail.com>
> Cc: Divya Gehlot <di...@gmail.com>, "user @spark" <
> user@spark.apache.org>
> Subject: Re: write and call UDF in spark dataframe
>
> Hi Divya,
>
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in
> dataframe DSL you can try like this,
>
> val new_df = df.select(from_unixtime($"time").as("newtime"))
>
>
> Thanks,
> Rishabh.
>
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
> dev.rabin.banerjee@gmail.com> wrote:
>
>> Hi Divya ,
>>
>> Try,
>>
>> val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr")
>>
>> Regards,
>> Rabin
>>
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts
>>> unix tme stamp to date tiime .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
Re: write and call UDF in spark dataframe
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Hi Divya
In general you will get better performance if you can minimize your use of
UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
treat your UDF as a block box.
Andy
From: Rishabh Bhardwaj <rb...@gmail.com>
Date: Wednesday, July 20, 2016 at 4:22 AM
To: Rabin Banerjee <de...@gmail.com>
Cc: Divya Gehlot <di...@gmail.com>, "user @spark"
<us...@spark.apache.org>
Subject: Re: write and call UDF in spark dataframe
> Hi Divya,
>
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in dataframe DSL
> you can try like this,
>
>> val new_df = df.select(from_unixtime($"time").as("newtime"))
>
> Thanks,
> Rishabh.
>
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <de...@gmail.com>
> wrote:
>> Hi Divya ,
>>
>> Try,
>>
>> val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from
>> mr")
>> Regards,
>> Rabin
>>
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts unix
>>> tme stamp to date tiime .
>>>
>>>
>>> Thanks,
>>> Divya
>>
>
Re: write and call UDF in spark dataframe
Posted by Rishabh Bhardwaj <rb...@gmail.com>.
Hi Divya,
There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
Rabin has used that in the sql query,if you want to use it in dataframe DSL
you can try like this,
val new_df = df.select(from_unixtime($"time").as("newtime"))
Thanks,
Rishabh.
On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
dev.rabin.banerjee@gmail.com> wrote:
> Hi Divya ,
>
> Try,
>
> val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr")
>
> Regards,
> Rabin
>
> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> Could somebody share example of writing and calling udf which converts
>> unix tme stamp to date tiime .
>>
>>
>> Thanks,
>> Divya
>>
>
>
Re: write and call UDF in spark dataframe
Posted by Rabin Banerjee <de...@gmail.com>.
Hi Divya ,
Try,
val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr")
Regards,
Rabin
On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> Could somebody share example of writing and calling udf which converts
> unix tme stamp to date tiime .
>
>
> Thanks,
> Divya
>