You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2026/01/06 02:16:45 UTC

[I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

2010YOUY01 opened a new issue, #19658:
URL: https://github.com/apache/datafusion/issues/19658

   ### Is your feature request related to a problem or challenge?
   
   This is a follow on work for https://github.com/apache/datafusion/pull/19626
   
   The above PR implemented `BaselineMetrics` tracking for `AsyncFuncExec`, except  `elapsed_compute`, the reason is we have found it requires some additional change to make it work as expected, see the original discussion for details: https://github.com/apache/datafusion/pull/19626#discussion_r2660989272
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3721888546

   I'm not sure who might know this better, or we could leave this ticket for someone who has real usage for this feature ๐Ÿค” , if thatโ€™s not the case for you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "feniljain (via GitHub)" <gi...@apache.org>.
feniljain commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3722230685

   Sure, that makes more sense! Will unassign myself :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "feniljain (via GitHub)" <gi...@apache.org>.
feniljain commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3722230928

   untake


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "feniljain (via GitHub)" <gi...@apache.org>.
feniljain commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3713160826

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "tschwarzinger (via GitHub)" <gi...@apache.org>.
tschwarzinger commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-4646384824

   Is there anything special about how `AsyncFuncExec` works? I've just stumbled across this issue and thought whether simply wrapping the future returned by `invoke_with_args` using something like this:
   
   ```rust
   /// A wrapper that measures the CPU time spent actively polling.
   struct MeasurePoll<F> {
       inner: Pin<Box<F>>,
       time_metric: Time,
   }
   
   impl<F: Future> Future for MeasurePoll<F> {
       type Output = F::Output;
   
       fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
           let cloned_inner = self.time_metric.clone();
           let handle = cloned_inner.timer();
           let result = self.inner.as_mut().poll(cx);
           drop(handle);
           result
       }
   }
   ```
   
   The intent of this wrapper is to stop the timer every time `Poll::Pending` is being returned.
   
   We've used it like this but we have not tested this thoroughly so there might be an issue with this approach.
   
   ```rust
   let physical_plan_future = state
           .query_planner()
           .create_physical_plan(&optimized_plan, &state);
       let physical_plan = MeasurePoll {
           inner: Box::pin(physical_plan_future),
           time_metric: planning_compute.clone(),
       }
       .await?;
   
       let planning_compute_nanos =
           u64::try_from(planning_compute.value()).unwrap_or(u64::MAX);
       let explanation = QueryExplanation {
           planning_latency: planning_time_start.elapsed(),
           planning_compute: Duration::from_nanos(planning_compute_nanos),
           initial_logical_plan: logical_plan,
           optimized_logical_plan: optimized_plan,
           execution_plan: Arc::clone(&physical_plan),
       };
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3721700483

   Perhaps we can keep the existing API, and add an optional new API `invoke_with_timer()`, and async UDFs can optionally migrate to a compute time tracked version manually


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "feniljain (via GitHub)" <gi...@apache.org>.
feniljain commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3718972380

   Hey @2010YOUY01 ๐Ÿ‘‹๐Ÿป
   
   I have been exploring this ticket, and I believe this would require a change to `AsyncScalarUDFImpl` itself? At the end of the chain, `invoke_async_with_args` is an async function too, so any driver code would `await` on it, and we don't want to include the polling time in our metrics, which means we would have to change the API of `invoke_async_with_args` to pass in a timer which user would have to manually start and stop in their implementation. If they don't, they would miss out on time.
   
   I currently don't see any other way to force the user to make use of that. For measuring time spent in our driver code, I believe we can pass the timer around and keep taking measurements using [intermediate](https://github.com/apache/datafusion/blob/566bcde9e12992932f684fe6509de39ae78cce9f/datafusion/physical-expr-common/src/metrics/baseline.rs#L103-L111) method.
   
   What do you think? Am I missing something? If we need to change the API, what according to you would be a good way to migrate all downstream consumers to the same?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Track `elapsed_compute` for `AsyncFuncExec` [datafusion]

Posted by "feniljain (via GitHub)" <gi...@apache.org>.
feniljain commented on issue #19658:
URL: https://github.com/apache/datafusion/issues/19658#issuecomment-3721835486

   Yup, that's one way! In general, as it seems to be a bigger change, do you think we should involve someone else in the discussion too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org