Looking for SystemDS UDFs which are SystemML equivalent

[Sagar Simon Gonsalves <sg...@ibm.com.INVALID>]

Hi,
I’m working on moving our code base from SystemML 1.2.0 to SystemDS 3.2.0,

And for that I’m looking for three UDFs from SystemML that I couldn’t find in SystemDS.

The UDF equivalent  that I’m looking for in SystemDS are:

  1.  org.apache.sysml.udf.lib.RowClassMeet
  2.  org.apache.sysml.udf.lib.CumSumProd
  3.  org.apache.sysml.udf.lib.MultiInputCbind

Can you please let me know if SystemDS has any equivalent functions or UDFs available?

Thanks,

Sagar Simon Gonsalves
Software Developer
Silicon Valley Lab

IBM

Re: Looking for SystemDS UDFs which are SystemML equivalent

[Matthias Boehm <mb...@gmail.com>]

thanks for reaching out Sagar. Let me first give the context and 
subsequently some proposals to implement these things:

a) Context: When we removed the MR backend, in favor of Spark and new 
backends, we also removed the UDF framework because it was only 
implemented for MR/in-memory UDFs. Originally, we wanted to build a new 
UDF framework (for Spark and other backends) but then realized there are 
better ways via DML-bodied builtin functions, new primitives, and simple 
custom lambda functions.

b) Alternatives:
* MultiInputCbind: this was mostly a performance feature for avoiding 
allocations of "cbind(cbind(cbind(X1,X2),X3),X4)", but we now have 
rewrites to internally rewrite such sequence to an nary 
cbind(X1,X2,X3,X4) (and rbind equivalently) anyway. So one could either 
write above sequence, or directly call the nary cbind via 
"cbind(list(X1,X2,X3,X4))".

* CumSumProd: this functionality is meanwhile directly supported via 
cumsumprod(Y) where Y=cbind(X,C). There is also a joint paper with 
previous SystemML team members from IBM on it:
https://mboehm7.github.io/resources/btw2019.pdf

* RowClassMeet: This one is a bit tricky. First I thought I should 
recommend our new lambda UDF functions (for custom cell/row/column 
operations) via "map(F, "x -> ...", margin=1)", but I think the easiest 
for you would be if we add this as a directly supported multi-return 
function (like eigen()). I'll reintegrate this functionality into the 
repository tomorrow morning.

Regards,
Matthias

On 6/5/2024 7:18 AM, Sagar Simon Gonsalves wrote:
> Hi,
> I’m working on moving our code base from SystemML 1.2.0 to SystemDS 3.2.0,
>
> And for that I’m looking for three UDFs from SystemML that I couldn’t find in SystemDS.
>
> The UDF equivalent  that I’m looking for in SystemDS are:
>
>    1.  org.apache.sysml.udf.lib.RowClassMeet
>    2.  org.apache.sysml.udf.lib.CumSumProd
>    3.  org.apache.sysml.udf.lib.MultiInputCbind
>
> Can you please let me know if SystemDS has any equivalent functions or UDFs available?
>
> Thanks,
>
> Sagar Simon Gonsalves
> Software Developer
> Silicon Valley Lab
>
> IBM
>