You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "marin-ma (via GitHub)" <gi...@apache.org> on 2024/08/26 09:13:07 UTC
[I] [VL] Remove native udf registration [incubator-gluten]
marin-ma opened a new issue, #7015:
URL: https://github.com/apache/incubator-gluten/issues/7015
### Description
Currently, the loaded native udf will be registered into Spark's FunctionRegistry to make the udf pass SQL analysis. However, the registration is opaque to the users, also it's not the way to use Java UDF in spark. The native udf framework should be improved to only allow the native udf usable by explicitly registering the function through `CREATE TEMPORARY FUNCTION`. This approach not only expose the udf registration process to users, it also guarantee there's an available Java version UDF in case of fallback.
At runtime, native UDFs should be registered alongside their Java implementations via `CREATE TEMPORARY FUNCTION`.
Once registered, Gluten can parse and offload these UDFs to Velox during execution,
meanwhile ensuring proper fallback to Java UDFs when necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Remove native udf registration [incubator-gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on issue #7015:
URL: https://github.com/apache/incubator-gluten/issues/7015#issuecomment-2316645200
@FelixYBW @marin-ma Thanks, we use Hive UDFS, and have `CREATE TEMPORARY FUNCTION` in our ETL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Remove native udf registration [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma closed issue #7015: [VL] Remove native udf registration
URL: https://github.com/apache/incubator-gluten/issues/7015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Remove native udf registration [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #7015:
URL: https://github.com/apache/incubator-gluten/issues/7015#issuecomment-2309741710
This change may affect your existing workloads. @kecookier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Remove native udf registration [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7015:
URL: https://github.com/apache/incubator-gluten/issues/7015#issuecomment-2313101540
@kecookier did you use Hive UDFs or Spark UDFs? for Hive UDFs, the SQL must have ` CREATE TEMPORARY FUNCTION ` statement. The function will be registered there. So we needn't register it in native.
To Spark functions, we still don't have a good way to catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Remove native udf registration [incubator-gluten]
Posted by "kecookier (via GitHub)" <gi...@apache.org>.
kecookier commented on issue #7015:
URL: https://github.com/apache/incubator-gluten/issues/7015#issuecomment-2316647190
cc @NEUpanning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org