You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "geoffreyclaude (via GitHub)" <gi...@apache.org> on 2025/02/07 17:38:38 UTC

[PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

geoffreyclaude opened a new pull request, #14547:
URL: https://github.com/apache/datafusion/pull/14547

   ## Which issue does this PR close?
   
   Relates to #9415. Does not fully close the issue, but moves forward with a pre-requisite.
   
   ## Rationale for this change
   
   This allows DataFusion to integrate with users of the [`tracing`](https://docs.rs/tracing/latest/tracing/) crate by propagating the trace context as users would expect, without investing in the full integration of the `tracing` ecosystem.
   When the (new) `tracing` feature is enabled, all tasks spawned on new threads (e.g. those spawned during repartitioning or while reading/writing Parquet files) inherit the current tracing span. This enhancement allows to propagate trace context through thread boundaries, into external data sources or custom exec nodes, and allows linking all generated logs and spans to the expected trace context.
   Previously, tasks spawned on new threads would lose the trace context, as it is thread-local and must be "manually" propagated to the new thread.
   
   ## What changes are included in this PR?
   
   - Update the common runtime so that tasks spawned on new threads are instrumented with the current tracing span when the `tracing` feature is enabled by wrapping the `tokio::task::JoinSet` in a custom `JoinSet`.
   - Add a new Cargo.toml feature (`tracing`) in the common-runtime crate, along with necessary dependency updates.
   - Provide an integration example in `datafusion-examples/examples/tracing.rs` that runs a SQL query over the `alltypes_tiny_pages_plain.parquet` file to demonstrate end-to-end propagation of the tracing context across multiple threads.
   - Update root `README.md` to reflect the availability and usage of the new `trace` feature.
   
   ## Are these changes tested?
   
   Yes. While there are no dedicated unit tests for this feature, the integration example in `datafusion-examples/examples/tracing.rs` serves as a comprehensive test. This example executes a query that triggers task spawns (such as through repartitioning and Parquet reading) and logs tracing output. By reviewing the logs, one can verify that the tracing span context is correctly propagated end to end.
   
   ## Are there any user-facing changes?
   
   No changes are expected for users who do not enable the `tracing` feature. The performance overhead *should* be inexistent when the feature is disabled, and completely negligible when enabled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2738119213

   (this is a very nice first contribution @geoffreyclaude -- thank you for sticking with it 🙏 )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on code in PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#discussion_r1954913978


##########
datafusion/common-runtime/src/join_set.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::future::Future;
+use std::task::{Context, Poll};
+use tokio::runtime::Handle;
+use tokio::task::JoinSet as TokioJoinSet;
+use tokio::task::{AbortHandle, Id, JoinError, LocalSet};
+
+#[cfg(feature = "tracing")]
+mod trace_utils {
+    use std::future::Future;
+    use tracing_futures::Instrument;
+
+    /// Instruments the provided future with the current tracing span.
+    pub fn trace_future<F, T>(future: F) -> impl Future<Output = T> + Send
+    where
+        F: Future<Output = T> + Send + 'static,
+        T: Send,
+    {
+        future.in_current_span()
+    }
+
+    /// Wraps the provided blocking function to execute within the current tracing span.
+    pub fn trace_block<F, T>(f: F) -> impl FnOnce() -> T + Send + 'static
+    where
+        F: FnOnce() -> T + Send + 'static,
+        T: Send,
+    {
+        let span = tracing::Span::current();
+        move || span.in_scope(f)
+    }
+}
+
+/// A wrapper around Tokio's [`JoinSet`](tokio::task::JoinSet) that transparently forwards all public API calls
+/// while optionally instrumenting spawned tasks and blocking closures with the current tracing span
+/// when the `trace` feature is enabled.
+#[derive(Debug)]
+pub struct JoinSet<T> {
+    inner: TokioJoinSet<T>,
+}
+
+impl<T> Default for JoinSet<T> {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl<T> JoinSet<T> {
+    /// [JoinSet::new](tokio::task::JoinSet::new) - Create a new JoinSet.
+    pub fn new() -> Self {
+        Self {
+            inner: TokioJoinSet::new(),
+        }
+    }
+
+    /// [JoinSet::len](tokio::task::JoinSet::len) - Return the number of tasks.
+    pub fn len(&self) -> usize {
+        self.inner.len()
+    }
+
+    /// [JoinSet::is_empty](tokio::task::JoinSet::is_empty) - Check if the JoinSet is empty.
+    pub fn is_empty(&self) -> bool {
+        self.inner.is_empty()
+    }
+}
+impl<T: 'static> JoinSet<T> {
+    /// [JoinSet::spawn](tokio::task::JoinSet::spawn) - Spawn a new task.
+    pub fn spawn<F>(&mut self, task: F) -> AbortHandle

Review Comment:
   Question: All public stable methods of the `tokio::task::JoinSet` are wrapped. Should only the methods used in DataFusion be wrapped for conciseness?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2729588907

   Thanks @geoffreyclaude  -- I will try and look at this.
   
   I think @goldmedal  was also interested in this capability (using tracing via the `otel` crate -- I will try and find time to look at this shortly. I am really sorry for the delay


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on code in PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#discussion_r1976361324


##########
datafusion/common-runtime/src/join_set.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::future::Future;
+use std::task::{Context, Poll};
+use tokio::runtime::Handle;
+use tokio::task::JoinSet as TokioJoinSet;
+use tokio::task::{AbortHandle, Id, JoinError, LocalSet};
+
+#[cfg(feature = "tracing")]
+mod trace_utils {
+    use std::future::Future;
+    use tracing_futures::Instrument;
+
+    /// Instruments the provided future with the current tracing span.
+    pub fn trace_future<F, T>(future: F) -> impl Future<Output = T> + Send
+    where
+        F: Future<Output = T> + Send + 'static,
+        T: Send,
+    {
+        future.in_current_span()
+    }
+
+    /// Wraps the provided blocking function to execute within the current tracing span.
+    pub fn trace_block<F, T>(f: F) -> impl FnOnce() -> T + Send + 'static
+    where
+        F: FnOnce() -> T + Send + 'static,
+        T: Send,
+    {
+        let span = tracing::Span::current();
+        move || span.in_scope(f)
+    }
+}
+
+/// A wrapper around Tokio's [`JoinSet`](tokio::task::JoinSet) that transparently forwards all public API calls
+/// while optionally instrumenting spawned tasks and blocking closures with the current tracing span
+/// when the `trace` feature is enabled.

Review Comment:
   `trace` -> `tracing`



##########
datafusion/common-runtime/src/common.rs:
##########
@@ -36,6 +37,7 @@ impl<R: 'static> SpawnedTask<R> {
         R: Send,
     {
         let mut inner = JoinSet::new();
+
         inner.spawn(task);

Review Comment:
   Remove extra line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: introduce `JoinSetTracer` trait for tracing context propagation in spawned tasks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #14547:
URL: https://github.com/apache/datafusion/pull/14547


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2739859432

   @alamb and @goldmedal thanks for the review!
   
   > From my perspective this PR now closes that issue's stated request:
   > 
   > > I would like to make it easy for people to add DataFusion ExecutionPlan level tracing to their systems as well.
   > 
   > Is it ok if we close it as well after this is merged?
   
   Ok for me at least! There'll probably be some minor adjustments to make once we start playing with it (maybe some other futures to wrap, for instance in `ListingTable::list_files_for_scan` the `future::try_join_all` and `StreamExt::buffer_unordered` might cause `tokio` to schedule on separate threads), but I think the core problem is solved for ExecutionPath tracing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2729811174

   I'll take a look this PR in a few day.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on code in PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#discussion_r1976361324


##########
datafusion/common-runtime/src/join_set.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::future::Future;
+use std::task::{Context, Poll};
+use tokio::runtime::Handle;
+use tokio::task::JoinSet as TokioJoinSet;
+use tokio::task::{AbortHandle, Id, JoinError, LocalSet};
+
+#[cfg(feature = "tracing")]
+mod trace_utils {
+    use std::future::Future;
+    use tracing_futures::Instrument;
+
+    /// Instruments the provided future with the current tracing span.
+    pub fn trace_future<F, T>(future: F) -> impl Future<Output = T> + Send
+    where
+        F: Future<Output = T> + Send + 'static,
+        T: Send,
+    {
+        future.in_current_span()
+    }
+
+    /// Wraps the provided blocking function to execute within the current tracing span.
+    pub fn trace_block<F, T>(f: F) -> impl FnOnce() -> T + Send + 'static
+    where
+        F: FnOnce() -> T + Send + 'static,
+        T: Send,
+    {
+        let span = tracing::Span::current();
+        move || span.in_scope(f)
+    }
+}
+
+/// A wrapper around Tokio's [`JoinSet`](tokio::task::JoinSet) that transparently forwards all public API calls
+/// while optionally instrumenting spawned tasks and blocking closures with the current tracing span
+/// when the `trace` feature is enabled.

Review Comment:
   `trace` -> `tracing`



##########
datafusion/common-runtime/src/common.rs:
##########
@@ -36,6 +37,7 @@ impl<R: 'static> SpawnedTask<R> {
         R: Send,
     {
         let mut inner = JoinSet::new();
+
         inner.spawn(task);

Review Comment:
   Remove extra line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#discussion_r2003274670


##########
datafusion/common-runtime/src/join_set.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::future::Future;
+use std::task::{Context, Poll};
+use tokio::runtime::Handle;
+use tokio::task::JoinSet as TokioJoinSet;
+use tokio::task::{AbortHandle, Id, JoinError, LocalSet};
+
+#[cfg(feature = "tracing")]
+mod trace_utils {
+    use std::future::Future;
+    use tracing_futures::Instrument;
+
+    /// Instruments the provided future with the current tracing span.
+    pub fn trace_future<F, T>(future: F) -> impl Future<Output = T> + Send
+    where
+        F: Future<Output = T> + Send + 'static,
+        T: Send,
+    {
+        future.in_current_span()
+    }
+
+    /// Wraps the provided blocking function to execute within the current tracing span.
+    pub fn trace_block<F, T>(f: F) -> impl FnOnce() -> T + Send + 'static
+    where
+        F: FnOnce() -> T + Send + 'static,
+        T: Send,
+    {
+        let span = tracing::Span::current();
+        move || span.in_scope(f)
+    }
+}
+
+/// A wrapper around Tokio's [`JoinSet`](tokio::task::JoinSet) that transparently forwards all public API calls
+/// while optionally instrumenting spawned tasks and blocking closures with the current tracing span
+/// when the `trace` feature is enabled.
+#[derive(Debug)]
+pub struct JoinSet<T> {
+    inner: TokioJoinSet<T>,
+}
+
+impl<T> Default for JoinSet<T> {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl<T> JoinSet<T> {
+    /// [JoinSet::new](tokio::task::JoinSet::new) - Create a new JoinSet.
+    pub fn new() -> Self {
+        Self {
+            inner: TokioJoinSet::new(),
+        }
+    }
+
+    /// [JoinSet::len](tokio::task::JoinSet::len) - Return the number of tasks.
+    pub fn len(&self) -> usize {
+        self.inner.len()
+    }
+
+    /// [JoinSet::is_empty](tokio::task::JoinSet::is_empty) - Check if the JoinSet is empty.
+    pub fn is_empty(&self) -> bool {
+        self.inner.is_empty()
+    }
+}
+impl<T: 'static> JoinSet<T> {
+    /// [JoinSet::spawn](tokio::task::JoinSet::spawn) - Spawn a new task.
+    pub fn spawn<F>(&mut self, task: F) -> AbortHandle

Review Comment:
   Wrapping all methods makes sense to me. The developer would expect using this wrapper to be the same as the original one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2669082144

   @alamb and @erratic-pattern 
   What do you think about this proposal PR as the first step to support tracing spans? DataDog uses traces a lot and we are planning to contribute to fully support this feature


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2697021679

   @alamb: I've gone ahead and refactored to allow "injecting" the tracing behavior at runtime. As predicted, the code is a bit scary looking, especially due to the Box/Unbox dance needed to get static Sized closures.
   
   But it works (see `datafusion-examples/examples/tracing.rs`) without needing the `tracing` crate (optionally or not) inside `datafusion-common-runtime`!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on code in PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#discussion_r1954912733


##########
README.md:
##########
@@ -126,6 +126,7 @@ Optional features:
 - `backtrace`: include backtrace information in error messages
 - `pyarrow`: conversions between PyArrow and DataFusion types
 - `serde`: enable arrow-schema's `serde` feature
+- `tracing`: propagates the current span across thread boundaries

Review Comment:
   Question: Is the new `tracing` feature required? Or should we just make it the default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2676860682

   Hi @geoffreyclaude @NGA-TRAN -- I will plan to review this PR shortly. Sorry for the delay. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2694747715

   > Thank you for working on this @geoffreyclaude -- I am sorry for the delay in responding.
   > 
   > Primarily my concern about this PR is that it adds new more dependencies (albiet optional ones) to DataFusion. We already spend a non trivial amount of maintenance time keeping up with dependencies / fixing related issues, etc .
   > 
   > I also worry that not everyone uses the tracing crate.
   > 
   > New features also increase the compile time, such as
   > 
   > * [Build time regression #14256](https://github.com/apache/datafusion/issues/14256)
   > 
   > > the "TracingExec", but it's a pretty wonky workaround. It also fails when the source nodes of the plan spawn new tasks, as in the ParquetSink, which prevent the >underlying object_store from linking its spans to the main trace.
   > 
   > FWIW the way we have handled this is to pass a tracing span along in PartitionedFile::extensions, but that is non ideal
   > 
   > https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.PartitionedFile.html#structfield.extensions
   > 
   > ## Questions / Suggestions
   > Is there some way to avoid adding a direct dependency on `tracing` but still keep the API flexibility you need?
   > 
   > For example, perhaps we could create a different `JoinSet` that instead of hard coding `tracing` could provide the callbacks for spawning tasks 🤔
   
   And thank you @alamb for the detailed response! I'll try to address your points one by one.
   
   > I also worry that not everyone uses the tracing crate.
   the `tracing` crate has pretty much become the standard crate for instrumentation, same as how `tokio` is the standard crate for async runtimes. Almost everyone who's interested in instrumenting their DataFusion execution will be using the `tracing` crate. Moreover, if we push this further and deliver a full execution plan instrumentation solution that relies on `tracing` (as a separate contrib crate,) it will probably become the de-facto DataFusion standard.
   
   > New features also increase the compile time
   This is why I added a feature guard, as tracing definitely adds some compilation overhead. With the feature disabled, there should be no impact at all.
   
   > pass a tracing span along in PartitionedFile::extensions
   This works if you want to link the object store calls to the root span, but I believe it won't allow you to link them to the "real" parent span, the `DataSinkExec` that wraps the `ParquetSink`. It also doesn't allow you to instrument native and custom execution nodes past a `RepartitionExec` that spawns tasks on separate threads.
   
   > Is there some way to avoid adding a direct dependency on `tracing` but still keep the API flexibility you need?
   I really like that idea, as it doesn't need to pull in additional dependencies, and let's users chose their instrumentation solution.
   Unfortunately, if we want to dynamically "inject" the new tracing behavior at runtime, it's particularly tricky, in particular because:
   1. it should be a static injection, so each new `JoinSet` created with `JoinSet<T>::new()` picks it up
   2. it needs to work for all the generic types of `JoinSet`, from `JoinSet<()>` to `JoinSet<std::result::Result<usize, DataFusionError>>`. (So we probably need complex macros, or a hard-coded list of all the possible types.)
   3. It needs to be done once and once only, globally, without relying on `unsafe`.
   It's of course possible, but the additional complexity just doesn't seem worth it. ... Or maybe there's a super obvious way to do it which I completely missed!
   
   To me, I don't see a simpler way to open up full instrumentation of execution plans than what's done in this PR. It's super unfortunate that the `tracing` crate has a thread-local only storage for its context, but it doesn't seem likely to move away from this implementation decision. 
   
   Since the minimal change is pretty small, we can keep our fork of the change internally. But it also seemed like a good reason to push it upstream and contribute the full execution plan instrumentation code on top of it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: introduce `JoinSetTracer` trait for tracing context propagation in spawned tasks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14547:
URL: https://github.com/apache/datafusion/pull/14547#issuecomment-2740784949

   I also added this feature to the list of things we should document with the next release
   - https://github.com/apache/datafusion/issues/15072
   
   Thanks again @geoffreyclaude and @goldmedal 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org