[I] Add metrics and logging to JVM-scalar-UDF dispatch path [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #4176:
URL: https://github.com/apache/datafusion-comet/issues/4176

   ## Describe the problem
   
   The JVM-scalar-UDF dispatch path (on the prototype branch `prototype-jvm-scalar-udf`) is invisible at runtime:
   
   - No SQL metric distinguishes "rlike ran on the JVM path" from "rlike ran natively".
   - No log line on first dispatch per executor identifies which UDF is being routed and which classloader resolved it.
   - `EXPLAIN` output for `JvmScalarUdfExpr` is not surfaced in Spark's plan summary.
   
   This makes "why is rlike slow in prod?" or "why is the JVM path being chosen?" unanswerable without source-level debugging.
   
   ## Describe the potential solution
   
   Add SQL metrics on the parent `CometNativeExec` (or a dedicated metric collector keyed by UDF class name):
   
   - `numJvmUdfBatches` — counter, batches dispatched
   - `jvmUdfElapsedNanos` — accumulator, time spent in JNI + UDF body
   - `jvmUdfBytesIn` / `jvmUdfBytesOut` — optional, for transfer-cost visibility
   
   And add an `info`-level log per `(executor, UDF class name)` on first dispatch, naming the resolved class and the classloader that resolved it.
   
   Optionally, surface `JvmScalarUdf(<class_name>)` as the operator name in EXPLAIN by hooking the appropriate Comet plan-formatter site.
   
   ## Additional context
   
   Identified during code review of the JVM-scalar-UDF prototype. Filed as a follow-up so the prototype PR can ship without a metrics-design decision.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Add metrics and logging to JVM-scalar-UDF dispatch path [datafusion-comet]

["mizulun (via GitHub)" <gi...@apache.org>]

mizulun commented on issue #4176:
URL: https://github.com/apache/datafusion-comet/issues/4176#issuecomment-5756625656

   Hi @andygrove 
   #4891 seems to have addressed the EXPLAIN visibility part, while #5282 now propagates the task ClassLoader and #6048 added JVM Arrow-memory tracing. Is the remaining scope for this issue still the per-dispatch SQL metrics (numJvmUdfBatches, elapsed time, possibly bytes in/out) and first-dispatch logging?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org