You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Matt Hawes <ha...@gmail.com> on 2022/06/28 03:42:05 UTC
Javascript Based UDFs
Hi all, I'm thinking about trying to implement the ability to write spark
UDFs using javascript.
For the use case I have in mind, a lot of the code is already written in
javascript and so it would be very convenient to be able to call this
directly from spark.
I wanted to post here first before I start digging into the UDF code to see
if anyone has attempted this already or if people have thoughts on it. I
couldn't find anything in the Jira. I'd be especially appreciative of any
pointers towards relevant sections of the code to get started!
My rough plan is to do something similar to how python UDFs work (as I
understand them). I.e. call out to a javascript process, potentially just
something in GraalJs for example: https://github.com/oracle/graaljs.
I understand that there's probably a long discussion to be had here with
regards to making this part of Spark core, but I wanted to start that
discussion. :)
Best,
Matt
Re: Javascript Based UDFs
Posted by Sean Owen <sr...@gmail.com>.
I don't know how these frameworks work, but I'd hope that it takes
JavaScript and gives you some invokeable object. It's just Java executing
like anything else, nothing special, no special consideration for being in
a task? Why would it need another JVM?
On Mon, Jun 27, 2022, 11:18 PM Matt Hawes <ha...@gmail.com> wrote:
> Thanks for the reply! I had originally thought that this would incur a
> cost of spinning up a VM every time the UDF is called but thinking about it
> again you might be right. I guess if I make the VM accessible via a
> transient property on the UDF class then it would only be initialized once
> per executor right? Or would it be once per task?
>
> I also was worried that this would mean you end up paying a lot in SerDe
> cost if you send each row over to the VM one by one?
>
> On Mon, Jun 27, 2022 at 10:02 PM Sean Owen <sr...@gmail.com> wrote:
>
>> Rather than reimplement a new UDF, why not indeed just use an embedded
>> interpreter? if something can turn javascript into something executable you
>> can wrap that in a normal Java/Scala UDF and go.
>>
>> On Mon, Jun 27, 2022 at 10:42 PM Matt Hawes <ha...@gmail.com> wrote:
>>
>>> Hi all, I'm thinking about trying to implement the ability to write
>>> spark UDFs using javascript.
>>>
>>> For the use case I have in mind, a lot of the code is already written in
>>> javascript and so it would be very convenient to be able to call this
>>> directly from spark.
>>>
>>> I wanted to post here first before I start digging into the UDF code to
>>> see if anyone has attempted this already or if people have thoughts on it.
>>> I couldn't find anything in the Jira. I'd be especially appreciative of any
>>> pointers towards relevant sections of the code to get started!
>>>
>>> My rough plan is to do something similar to how python UDFs work (as I
>>> understand them). I.e. call out to a javascript process, potentially just
>>> something in GraalJs for example: https://github.com/oracle/graaljs.
>>>
>>> I understand that there's probably a long discussion to be had here with
>>> regards to making this part of Spark core, but I wanted to start that
>>> discussion. :)
>>>
>>> Best,
>>> Matt
>>>
>>>
Re: Javascript Based UDFs
Posted by Matt Hawes <ha...@gmail.com>.
Thanks for the reply! I had originally thought that this would incur a cost
of spinning up a VM every time the UDF is called but thinking about it
again you might be right. I guess if I make the VM accessible via a
transient property on the UDF class then it would only be initialized once
per executor right? Or would it be once per task?
I also was worried that this would mean you end up paying a lot in SerDe
cost if you send each row over to the VM one by one?
On Mon, Jun 27, 2022 at 10:02 PM Sean Owen <sr...@gmail.com> wrote:
> Rather than reimplement a new UDF, why not indeed just use an embedded
> interpreter? if something can turn javascript into something executable you
> can wrap that in a normal Java/Scala UDF and go.
>
> On Mon, Jun 27, 2022 at 10:42 PM Matt Hawes <ha...@gmail.com> wrote:
>
>> Hi all, I'm thinking about trying to implement the ability to write spark
>> UDFs using javascript.
>>
>> For the use case I have in mind, a lot of the code is already written in
>> javascript and so it would be very convenient to be able to call this
>> directly from spark.
>>
>> I wanted to post here first before I start digging into the UDF code to
>> see if anyone has attempted this already or if people have thoughts on it.
>> I couldn't find anything in the Jira. I'd be especially appreciative of any
>> pointers towards relevant sections of the code to get started!
>>
>> My rough plan is to do something similar to how python UDFs work (as I
>> understand them). I.e. call out to a javascript process, potentially just
>> something in GraalJs for example: https://github.com/oracle/graaljs.
>>
>> I understand that there's probably a long discussion to be had here with
>> regards to making this part of Spark core, but I wanted to start that
>> discussion. :)
>>
>> Best,
>> Matt
>>
>>
Re: Javascript Based UDFs
Posted by Sean Owen <sr...@gmail.com>.
Rather than reimplement a new UDF, why not indeed just use an embedded
interpreter? if something can turn javascript into something executable you
can wrap that in a normal Java/Scala UDF and go.
On Mon, Jun 27, 2022 at 10:42 PM Matt Hawes <ha...@gmail.com> wrote:
> Hi all, I'm thinking about trying to implement the ability to write spark
> UDFs using javascript.
>
> For the use case I have in mind, a lot of the code is already written in
> javascript and so it would be very convenient to be able to call this
> directly from spark.
>
> I wanted to post here first before I start digging into the UDF code to
> see if anyone has attempted this already or if people have thoughts on it.
> I couldn't find anything in the Jira. I'd be especially appreciative of any
> pointers towards relevant sections of the code to get started!
>
> My rough plan is to do something similar to how python UDFs work (as I
> understand them). I.e. call out to a javascript process, potentially just
> something in GraalJs for example: https://github.com/oracle/graaljs.
>
> I understand that there's probably a long discussion to be had here with
> regards to making this part of Spark core, but I wanted to start that
> discussion. :)
>
> Best,
> Matt
>
>