You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayshrivastava (via GitHub)" <gi...@apache.org> on 2026/02/17 21:48:46 UTC

[PR] wip [datafusion]

jayshrivastava opened a new pull request, #20416:
URL: https://github.com/apache/datafusion/pull/20416

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava closed pull request #20416: proto: serialize and dedupe dynamic filters
URL: https://github.com/apache/datafusion/pull/20416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2829875181


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +467,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64
+    }

Review Comment:
   It sounds like all of these cases are valid? Especially since datafusion let's people implement their own data sources.
   
   1. Hash join create a dynamic filter, then the data source calls `Arc::clone(&filter)`. The outer arc has a count of 2, inner has a count of 1.
   2. Hash join create a dynamic filter, then the data source calls `reassign_expr_columns -> new_with_children` only. I'm pretty sure in this case, there are two outer Arcs, each with a count of 1. And they point to the same inner Arc, which has a count of 2.
   3. A combination of 1, then 2, and so on which causes `weak=2, strong=2` and whatever other combinations.
   
   I think this is the correct implementation of `is_used` (and is what is used in my branch). In all cases, at least one should be incremented.
   ```
   Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2830265699


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +467,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64
+    }

Review Comment:
   yes, that is correct, we should check for both because both are possible 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2897109583


##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   The only expression that gets transformed in `snapshot_physical_expr` is the ones that implement `snapshot()`, and atm the only one is `DynamicFilterPhysicalExpr`, therefore for all other expressions calling snapshot_physical_expr is a no-op iiuc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2956620097


##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -860,6 +860,12 @@ message PhysicalExprNode {
   // across serde roundtrips.
   optional uint64 expr_id = 30;
 
+  // For DynamicFilterPhysicalExpr, this identifies the shared inner state.
+  // Multiple expressions may have different expr_id values (different outer Arc wrappers)
+  // but the same dynamic_filter_inner_id (shared inner state).
+  // Used to reconstruct shared inner state during deserialization.
+  optional uint64 dynamic_filter_inner_id = 31;
+

Review Comment:
   Removed. Now it's an "internal expr id"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2828422104


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +243,99 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Reconstructs a [`DynamicFilterPhysicalExpr`] from a snapshot.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    #[doc(hidden)]
+    pub fn new_from_snapshot(snapshot: DynamicFilterSnapshot) -> Self {
+        let DynamicFilterSnapshot {
+            children,
+            remapped_children,
+            generation,
+            inner_expr,
+            is_complete,
+        } = snapshot;
+
+        let state = if is_complete {
+            FilterState::Complete { generation }
+        } else {
+            FilterState::InProgress { generation }
+        };
+        let (state_watch, _) = watch::channel(state);
+
+        Self {
+            children,
+            remapped_children,
+            inner: Arc::new(RwLock::new(Inner {
+                generation,
+                expr: inner_expr,
+                is_complete,
+            })),
+            state_watch,
+            data_type: Arc::new(RwLock::new(None)),
+            nullable: Arc::new(RwLock::new(None)),
+        }
+    }
+
+    /// Atomically captures all state needed for serialization into a [`DynamicFilterSnapshot`].

Review Comment:
   just an idea, instead of this could we do something like implement the From triat to convert between Expr and Snapshot



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2843240928


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -88,6 +88,69 @@ struct Inner {
     is_complete: bool,
 }
 
+/// An atomic snapshot of a [`DynamicFilterPhysicalExpr`] used to reconstruct the expression during
+/// serialization / deserialization.
+pub struct DynamicFilterSnapshot {

Review Comment:
   It seems like we end up exposing all of the internals anyway: if there was a way to return a `dyn Serializable` it'd be one thing, but the proto machinery still needs to know all of the fields to serialize.



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   I also agree that it's quite hacky to use pointer addresses.
   
   The main reason I went down this path is that (I hoped) it "just worked" for all expressions and avoided special casing `DynamicFilterPhysicalExpr` or polluting the APIs.
   
   If we were to add a unique identifier it would be unfortunate if every `PhysicalExpr` implementation had to do something to handle this change, most of them don't have internal mutability.
   
   Maybe a compromise / first step would be:
   
   ```rust
   trait PhysicalExpr {
      fn expr_id(Arc<Self>) -> u64 {
          // do pointer shennanigans
      }
   }
   
   impl PhysicalExpr for DynamicFilterPhysicalExpr {
     fn expr_id(Arc<Self>) -> u64 {
       self.id  // some randomly generated integer
     }
   }
   ```
   
   That way all physical expressions can be deduplicated / uniquely identified?
   
   Alternatively the method could be `Option<u64>` and the default impl returns `None` -> no deduplication for this expression.



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -88,6 +88,69 @@ struct Inner {
     is_complete: bool,
 }
 
+/// An atomic snapshot of a [`DynamicFilterPhysicalExpr`] used to reconstruct the expression during
+/// serialization / deserialization.
+pub struct DynamicFilterSnapshot {

Review Comment:
   I also think the name snapshot makes it sounds like this is what would be returned from `PhysicalExpr::snapshot`, but that is not the case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2834776060


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -88,6 +88,69 @@ struct Inner {
     is_complete: bool,
 }
 
+/// An atomic snapshot of a [`DynamicFilterPhysicalExpr`] used to reconstruct the expression during
+/// serialization / deserialization.
+pub struct DynamicFilterSnapshot {

Review Comment:
   Maybe we should just add the getters and setters for the internal fields...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2824714254


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -88,6 +88,69 @@ struct Inner {
     is_complete: bool,
 }
 
+/// An atomic snapshot of a [`DynamicFilterPhysicalExpr`] used to reconstruct the expression during
+/// serialization / deserialization.
+pub struct DynamicFilterSnapshot {

Review Comment:
   Added this type because I thought it looked cleaner / simpler than having getters on the `DynamicFilterPhysicalExpr` itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2830088900


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +243,99 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Reconstructs a [`DynamicFilterPhysicalExpr`] from a snapshot.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    #[doc(hidden)]
+    pub fn new_from_snapshot(snapshot: DynamicFilterSnapshot) -> Self {
+        let DynamicFilterSnapshot {
+            children,
+            remapped_children,
+            generation,
+            inner_expr,
+            is_complete,
+        } = snapshot;
+
+        let state = if is_complete {
+            FilterState::Complete { generation }
+        } else {
+            FilterState::InProgress { generation }
+        };
+        let (state_watch, _) = watch::channel(state);
+
+        Self {
+            children,
+            remapped_children,
+            inner: Arc::new(RwLock::new(Inner {
+                generation,
+                expr: inner_expr,
+                is_complete,
+            })),
+            state_watch,
+            data_type: Arc::new(RwLock::new(None)),
+            nullable: Arc::new(RwLock::new(None)),
+        }
+    }
+
+    /// Atomically captures all state needed for serialization into a [`DynamicFilterSnapshot`].

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851190685


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -88,6 +88,69 @@ struct Inner {
     is_complete: bool,
 }
 
+/// An atomic snapshot of a [`DynamicFilterPhysicalExpr`] used to reconstruct the expression during
+/// serialization / deserialization.
+pub struct DynamicFilterSnapshot {

Review Comment:
   Could be... you kind of have already this, just to an intermediate structure instead



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2956618760


##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   I've kept the `snapshot()` method, but I removed its usage in the serializer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2958092872


##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   Sorry for the absence.
   
   A bit of context: the idea of the `snapshot()` method was that it would (1) allow us to get a simplified predicate that `PruningPredicate` knows how to handle (so that we didn't have to add another downcast into `DynamicFilterPhysicalExpr`) and (2) give us an expression we can serialize across the wire.
   
   I agree it hasn't panned out well as a design and is a good candidate for removal.
   
   IMO one of the warts with DataFusion's serialization machinery is that generally there is no method on these traits for them to decide how to serialize themselves -> the serialization code has to try downcasting into every known type. In an ideal world `DynamicFilterPhysicalExpr` could decide how to serialize and deserialize itself.
   
   
   > What we want in this PR, is to serialize the actual `DynamicFilterPhysicalExpr`, with all it's details, so probably ` snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at all for any expression, and we should be able to treat the dynamic filter as just another serializable `PhysicalExpr`
   
   > I've kept the `snapshot()` method, but I removed its usage in the serializer.
   
   👍🏻 yep this was always the loose plan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#issuecomment-4356463736

   Closing in favor of https://github.com/apache/datafusion/pull/21807


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2834561437


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -438,16 +587,6 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
     fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         self.render(f, |expr, f| expr.fmt_sql(f))
     }
-
-    fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>> {

Review Comment:
   Makes sense. I've restored the snapshot method!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851892219


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   > The is_used() method returns bool if there is a strong count greater than one in any of their inner or outer arcs, but you might perfectly have a strong count greater than one just because of how you happen to lay out your Rust code, and the filter might still be unused (e.g., in optimization passes, the strong count can be 6 or 7 easily, and the filter is still unused)
   
   I agree it’s a bit hacky and can be tricky to work with (for example, what we saw in https://github.com/apache/datafusion/issues/19715). However, the DynamicFilter isn’t currently public in any of the producer nodes, so it shouldn’t be easy to clone it unintentionally. The only place it can be obtained externally is via the `handle_child_pushdown_...` API on the consumer side, which would imply the filter is actually being used. I might be missing something, but in what situations would optimization passes increase the Arc strong count without the filter actually being used? 
   
   Overall, though I agree we should look for a better long term solution for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2868272194


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +291,52 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety
+    ///
+    /// The dynamic filter should not be in use when calling this method, otherwise there
+    /// may be undefined behavior. Changing the inner state of a filter may do the following:
+    /// - transition the state to complete without notifying the watch
+    /// - cause a generation number to be emitted which is out of order
+    pub fn new_from_source(
+        self: &Arc<Self>,
+        source: &DynamicFilterPhysicalExpr,
+    ) -> Result<Self> {
+        // If there's any references to this filter or any watchers, we should not replace the
+        // inner state.
+        if self.is_used() {
+            return internal_err!(
+                "Cannot replace the inner state of a DynamicFilterPhysicalExpr that is in use"
+            );
+        };
+
+        Ok(Self {
+            children: self.children.clone(),
+            remapped_children: self.remapped_children.clone(),
+            inner: Arc::clone(&source.inner),
+            state_watch: self.state_watch.clone(),

Review Comment:
   More notes to self
   - `PhysicalExtensionCodec` methods should take a converter as arguments
   - Distributed datafusion needs to use converters, specifically the deduping one



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2829385055


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +243,99 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Reconstructs a [`DynamicFilterPhysicalExpr`] from a snapshot.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    #[doc(hidden)]
+    pub fn new_from_snapshot(snapshot: DynamicFilterSnapshot) -> Self {
+        let DynamicFilterSnapshot {
+            children,
+            remapped_children,
+            generation,
+            inner_expr,
+            is_complete,
+        } = snapshot;
+
+        let state = if is_complete {
+            FilterState::Complete { generation }
+        } else {
+            FilterState::InProgress { generation }
+        };
+        let (state_watch, _) = watch::channel(state);
+
+        Self {
+            children,
+            remapped_children,
+            inner: Arc::new(RwLock::new(Inner {
+                generation,
+                expr: inner_expr,
+                is_complete,
+            })),
+            state_watch,
+            data_type: Arc::new(RwLock::new(None)),
+            nullable: Arc::new(RwLock::new(None)),
+        }
+    }
+
+    /// Atomically captures all state needed for serialization into a [`DynamicFilterSnapshot`].
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    #[doc(hidden)]
+    pub fn current_snapshot(&self) -> DynamicFilterSnapshot {
+        let (generation, inner_expr, is_complete) = {
+            let inner = self.inner.read();
+            (inner.generation, Arc::clone(&inner.expr), inner.is_complete)
+        };
+        DynamicFilterSnapshot {
+            children: self.children.clone(),
+            remapped_children: self.remapped_children.clone(),
+            generation,
+            inner_expr,
+            is_complete,
+        }
+    }
+
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety
+    ///
+    /// The dynamic filter should not be in use when calling this method, otherwise there
+    /// may be undefined behavior. This method may do the following or worse:
+    /// - transition the state to complete without notifying the watch
+    /// - cause a generation number to be emitted which is out of order
+    pub fn new_from_source(&self, source: &DynamicFilterPhysicalExpr) -> Result<Self> {
+        // Best effort check that no one is subscribed.
+        if self.state_watch.receiver_count() > 0 {
+            return internal_err!(
+                "Cannot replace the inner state of a DynamicFilterPhysicalExpr that has subscribers"
+            );

Review Comment:
   there are suscribed consumers if someone calls `wait_complete` or `wait_update` however consumers are not forced to to call either of this methods, and actually most of consumers will never call them (like parquet). I think another way you can check if there are any consumers is probably doing something similar to is_used and checking the strong count of `inner' 
   



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +467,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64
+    }

Review Comment:
   you might be interested in looking at https://github.com/apache/datafusion/issues/19715#issuecomment-3733169909 we found out that for some cases the `inner` struct was not the one being cloned but the outer struct (because we never called `reassign_expr_columns -> new_with_children`. I have an example DataSource repro [here](https://github.com/LiaCastaneda/datafusion/commit/5dedd792670a5c0ea4be43e4ee3c88f355c5fc48) were the inner struct is not clonned.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851241245


##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3918,24 +3936,51 @@ impl PhysicalProtoConverterExtension for DeduplicatingDeserializer {
     where
         Self: Sized,
     {
-        if let Some(expr_id) = proto.expr_id {
-            // Check cache first
-            if let Some(cached) = self.cache.borrow().get(&expr_id) {
-                return Ok(Arc::clone(cached));
+        // The entire expr is cached, so re-use it.
+        if let Some(expr_id) = proto.expr_id
+            && let Some(cached) = self.cache.borrow().get(&expr_id)
+        {
+            return Ok(Arc::clone(cached));
+        }
+
+        // Cache miss, we must deserialize the expr.
+        let mut expr =
+            parse_physical_expr_with_converter(proto, ctx, input_schema, codec, self)?;
+
+        // Check if we need to share inner state with a cached dynamic filter
+        if let Some(dynamic_filter_id) = proto.dynamic_filter_inner_id {

Review Comment:
   The amount of special-casing for handling dynamic filters in the protobuf code seems too big in this PR.
   
   The fact that dynamic filters are in a situation where they claim to be normal `PhysicalExpr` but they anyway need special treatment and `if dyn_filter { do this special other thing }`  in several parts of the codebase makes me think that there might be better ways of approaching things in general. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2839651075


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -497,15 +646,23 @@ mod test {
             &filter_schema_1,
         )
         .unwrap();
-        let snap = dynamic_filter_1.snapshot().unwrap().unwrap();
-        insta::assert_snapshot!(format!("{snap:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);
+        let df1 = dynamic_filter_1
+            .as_any()
+            .downcast_ref::<DynamicFilterPhysicalExpr>()
+            .unwrap();
+        let current1 = df1.current().unwrap();

Review Comment:
   Can we move back on asserting on the snapshot()? since it would have the same outuput



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -497,15 +646,23 @@ mod test {
             &filter_schema_1,
         )
         .unwrap();
-        let snap = dynamic_filter_1.snapshot().unwrap().unwrap();
-        insta::assert_snapshot!(format!("{snap:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);
+        let df1 = dynamic_filter_1
+            .as_any()
+            .downcast_ref::<DynamicFilterPhysicalExpr>()
+            .unwrap();
+        let current1 = df1.current().unwrap();
+        insta::assert_snapshot!(format!("{current1:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);

Review Comment:
   Can we move back to asserting on `dynamic_filter.snapshot()` since I think they return the same output



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -497,15 +646,23 @@ mod test {
             &filter_schema_1,
         )
         .unwrap();
-        let snap = dynamic_filter_1.snapshot().unwrap().unwrap();
-        insta::assert_snapshot!(format!("{snap:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);
+        let df1 = dynamic_filter_1
+            .as_any()
+            .downcast_ref::<DynamicFilterPhysicalExpr>()
+            .unwrap();
+        let current1 = df1.current().unwrap();

Review Comment:
   Can we move back on asserting on the snapshot()? since it would have the same outuput



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2842625397


##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3818,6 +3822,18 @@ impl DeduplicatingSerializer {
             session_id: rand::random(),
         }
     }
+
+    fn hash(&self, ptr: u64) -> u64 {
+        // Hash session_id, pointer address, and process ID together to create expr_id.
+        // - session_id: random per serializer, prevents collisions when merging serializations
+        // - ptr: unique address per Arc within a process
+        // - pid: prevents collisions if serializer is shared across processes
+        let mut hasher = DefaultHasher::new();
+        self.session_id.hash(&mut hasher);
+        ptr.hash(&mut hasher);
+        std::process::id().hash(&mut hasher);
+        hasher.finish()
+    }
 }
 
 impl PhysicalProtoConverterExtension for DeduplicatingSerializer {

Review Comment:
   I suspect it may have been working because we were serializing the snapshot of the dynamic filter instead of the `DynamicFilterPhysicalExpr`. I don't think that's what we want. After changing that behavior to serialize/deserialize the `DynamicFilterPhysicalExpr`, then the `DeduplicatingSerializer` only dedupes the outer arc and not the inner.
   
   I did an investigation [here](https://github.com/datafusion-contrib/datafusion-distributed/issues/180#issuecomment-3902482676) with more details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2956637740


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   👋🏽 @adriangb @gabotechs @LiaCastaneda this PR is ready for another review. 
   
   I've attempted to create a better abstraction. I've put the details in the PR desc but will include them here too. Let me know what you think!
   
   > This PR aims to fix those problems by making a few changes 
   > 1. `PhysicalExpr::expr_id` is added. This is basically the arc pointer which we use today. Any expression can be deduped trivially using this ID
   > 2. For dynamic filter expressions, we now have a new trait `DedupablePhysicalExpr`. 
   >   - Its purpose is to allow `PhysicalExpr` to internally define what `dedupe` means. 
   >   - It adds a new concept, an `internal_expr_id` which is different than the one provided by `PhysicalExpr::expr_id`. It indicates that the expression cannot be deduped trivially
   >   - There's a `dedupe(self, other)` method which allows the implementor to define how to dedupe two expressions
   > 
   > Then we update the `DedupingSerializer` and `DedupingDeserializer` to dedupe on both levels, using the `PhysicalExpr::expr_id` and using the `internal_expr_id`.
   >  
   > One suspicious thing is that, since dynamic filters change with time (ie. have internal mutability unlike other `PhysicalExpr`), I thought it would be good to snapshot the generation as well. For this reason, `DedupablePhysicalExpr` has a concept of a `DedupeSnapshot` which provides the `internal_expr_id`. Both must be captured **atomically** to serialize the dynamic filter expr. 
   > 
   > This PR does not 
   > - add a test for the `HashJoinExec` and `DataSourceExec` filter pushdown case, but this is relevant follow up work.
   > - PhysicalExtensionCodec methods should take a converter as arguments. Distributed datafusion needs to use converters, specifically the deduping one
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2829446739


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +467,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64
+    }

Review Comment:
   you might be interested in looking at https://github.com/apache/datafusion/issues/19715#issuecomment-3733169909 we found out that for some cases the `inner` struct was not the one being cloned but the outer struct (because we never called `reassign_expr_columns -> new_with_children`. I have an example DataSource repro [here](https://github.com/LiaCastaneda/datafusion/commit/5dedd792670a5c0ea4be43e4ee3c88f355c5fc48) were the inner struct is not clonned. I still haven't looked in what context `inner_id` is used yet (I will continue reviewing tomorrow) but I wanted to mention it in case it helps.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2830332263


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -438,16 +587,6 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
     fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         self.render(f, |expr, f| expr.fmt_sql(f))
     }
-
-    fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>> {
-        // Return the current expression as a snapshot.
-        Ok(Some(self.current()?))
-    }
-
-    fn snapshot_generation(&self) -> u64 {
-        // Return the current generation of the expression.
-        self.inner.read().generation
-    }

Review Comment:
   I wonder if unimplementing these APIs would cause any regressions on any existing assumptions, for example the `ParquetOpener` [here](https://github.com/apache/datafusion/blob/0022d8e503f0dc0ee40ead545114147fc703e263/datafusion/datasource-parquet/src/opener.rs#L333) needs to know if a `PhysicalExpr` is dynamic by calling  `snapshot_generation`, if we don't implement this method then it will always return 0 by default and I think no predicates will be pushed down to parquet because `is_dynamic_physical_expr` would return false. 



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +291,52 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety
+    ///
+    /// The dynamic filter should not be in use when calling this method, otherwise there
+    /// may be undefined behavior. Changing the inner state of a filter may do the following:
+    /// - transition the state to complete without notifying the watch
+    /// - cause a generation number to be emitted which is out of order
+    pub fn new_from_source(
+        self: &Arc<Self>,
+        source: &DynamicFilterPhysicalExpr,
+    ) -> Result<Self> {
+        // If there's any references to this filter or any watchers, we should not replace the
+        // inner state.
+        if self.is_used() || self.state_watch.receiver_count() > 0 {

Review Comment:
   if there are watchers/consumers then` is_used()` will return `true`, so I think we can just check `is_used` here



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -438,16 +587,6 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
     fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         self.render(f, |expr, f| expr.fmt_sql(f))
     }
-
-    fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>> {

Review Comment:
   IIUC we removed `snapshot()` here because `snapshot_physical_expr` is called unconditionally at the top of `serialize_physical_expr_with_converter` -- if `snapshot()` returns` Some(self.current()?)`, the `DynamicFilterPhysicalExpr` gets replaced by its `inner` expression before the `downcast_ref::<DynamicFilterPhysicalExpr>()` branch ever gets a chance to see it. By returning None, the node passes through unchanged and gets serialized as a `whole` which is the behaviour we want.
   
   However, I wonder if a  safer approach would be to check for `DynamicFilterPhysicalExpr` before calling `snapshot_physical_expr` in `to_proto`, and only call `snapshot_physical_expr` if the expression is not a `DynamicFilterPhysicalExpr`. That way `snapshot()` could be restored to return `Some(self.current()?)`, which would also fix a regression for other callers that actually depend on snapshot() retuning the collapsed expression -- I think  [PruningPredicate::try_new](https://github.com/apache/datafusion/blob/0022d8e503f0dc0ee40ead545114147fc703e263/datafusion/pruning/src/pruning_predicate.rs#L464), calls `snapshot_physical_expr_opt` to transform a `DynamicFilterPhysicalExpr` into an expression it can analyze for pruning, so with the current approach I think that caller gets silently broken.



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +467,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64
+    }

Review Comment:
   yes, that is correct, we should check for both because both are possible



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851354781


##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   I see one of the main issues is how snapshotting works with dynamic filtering.
   
   In the physical expression trait:
   
   ```rust
       ...
       /// A system or function that can only deal with a hardcoded set of `PhysicalExpr` implementations
       /// or needs to serialize this state to bytes may not be able to handle these dynamic references.
       /// In such cases, we should return a simplified version of the `PhysicalExpr` that does not
       /// contain these dynamic references.
       ...
       fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>>;
   ```
   
   This doesn't return a snapshot of the `DynamicFilterPhysicalExpr`, it returns a snapshot to the inner computed physical expression, losing all the information of the original  `DynamicFilterPhysicalExpr`.
   
   What we want in this PR, is to serialize is the actual `DynamicFilterPhysicalExpr`, with all it's details, so probably ` snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at all for any expression, and we should be able to treat the dynamic filter as just another serializable `PhysicalExpr` 
   



##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   I see one of the main issues is how snapshotting works with dynamic filtering.
   
   In the physical expression trait:
   
   ```rust
       ...
       /// A system or function that can only deal with a hardcoded set of `PhysicalExpr` implementations
       /// or needs to serialize this state to bytes may not be able to handle these dynamic references.
       /// In such cases, we should return a simplified version of the `PhysicalExpr` that does not
       /// contain these dynamic references.
       ...
       fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>>;
   ```
   
   This doesn't return a snapshot of the `DynamicFilterPhysicalExpr`, it returns a snapshot to the inner computed physical expression, losing all the information of the original  `DynamicFilterPhysicalExpr`.
   
   What we want in this PR, is to serialize the actual `DynamicFilterPhysicalExpr`, with all it's details, so probably ` snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at all for any expression, and we should be able to treat the dynamic filter as just another serializable `PhysicalExpr` 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851241245


##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3918,24 +3936,51 @@ impl PhysicalProtoConverterExtension for DeduplicatingDeserializer {
     where
         Self: Sized,
     {
-        if let Some(expr_id) = proto.expr_id {
-            // Check cache first
-            if let Some(cached) = self.cache.borrow().get(&expr_id) {
-                return Ok(Arc::clone(cached));
+        // The entire expr is cached, so re-use it.
+        if let Some(expr_id) = proto.expr_id
+            && let Some(cached) = self.cache.borrow().get(&expr_id)
+        {
+            return Ok(Arc::clone(cached));
+        }
+
+        // Cache miss, we must deserialize the expr.
+        let mut expr =
+            parse_physical_expr_with_converter(proto, ctx, input_schema, codec, self)?;
+
+        // Check if we need to share inner state with a cached dynamic filter
+        if let Some(dynamic_filter_id) = proto.dynamic_filter_inner_id {

Review Comment:
   The amount of special-casing for handling dynamic filters in the protobuf code seems too big in this PR.
   
   The fact that dynamic filters are in a situation where they claim to be normal `PhysicalExpr` but they anyway need special treatment in several parts of the codebase makes me think that there might be better ways of approaching things in general. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2897086408


##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   It looks like `snapshot()` is only implemented for `DynamicFilterPhysicalExpr` so maybe if we just remove the `snapshot_physical_expr` call in `serialize_physical_expr_with_converter` and handle `DynamicFilterPhysicalExpr` as a normal expression in the downcast chain it will just work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2834559974


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +291,52 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety
+    ///
+    /// The dynamic filter should not be in use when calling this method, otherwise there
+    /// may be undefined behavior. Changing the inner state of a filter may do the following:
+    /// - transition the state to complete without notifying the watch
+    /// - cause a generation number to be emitted which is out of order
+    pub fn new_from_source(
+        self: &Arc<Self>,
+        source: &DynamicFilterPhysicalExpr,
+    ) -> Result<Self> {
+        // If there's any references to this filter or any watchers, we should not replace the
+        // inner state.
+        if self.is_used() || self.state_watch.receiver_count() > 0 {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851286605


##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -860,6 +860,12 @@ message PhysicalExprNode {
   // across serde roundtrips.
   optional uint64 expr_id = 30;
 
+  // For DynamicFilterPhysicalExpr, this identifies the shared inner state.
+  // Multiple expressions may have different expr_id values (different outer Arc wrappers)
+  // but the same dynamic_filter_inner_id (shared inner state).
+  // Used to reconstruct shared inner state during deserialization.
+  optional uint64 dynamic_filter_inner_id = 31;
+

Review Comment:
   I think we should find ways of not leaking this detail here.



##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3918,24 +3936,51 @@ impl PhysicalProtoConverterExtension for DeduplicatingDeserializer {
     where
         Self: Sized,
     {
-        if let Some(expr_id) = proto.expr_id {
-            // Check cache first
-            if let Some(cached) = self.cache.borrow().get(&expr_id) {
-                return Ok(Arc::clone(cached));
+        // The entire expr is cached, so re-use it.
+        if let Some(expr_id) = proto.expr_id
+            && let Some(cached) = self.cache.borrow().get(&expr_id)
+        {
+            return Ok(Arc::clone(cached));
+        }
+
+        // Cache miss, we must deserialize the expr.
+        let mut expr =
+            parse_physical_expr_with_converter(proto, ctx, input_schema, codec, self)?;
+
+        // Check if we need to share inner state with a cached dynamic filter
+        if let Some(dynamic_filter_id) = proto.dynamic_filter_inner_id {

Review Comment:
   The amount of special-casing for handling dynamic filters in the protobuf code seems to big in this PR.
   
   The fact that dynamic filters are in a situation where they claim to be normal `PhysicalExpr` but they anyway need special treatment in several parts of the codebase makes me think that there might be better ways of approaching things in general. 



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   Overall, I think the current way dynamic filters are playing with raw pointer addresses is a bit like playing with fire.
   
   For example:
   - The `is_used()` method returns bool if there is a strong count greater than one in any of their inner or outer arcs, but you might perfectly have a strong count greater than one just because of how you happen to lay out your Rust code, and the filter might still be unused (e.g., in optimization passes, the strong count can be 6 or 7 easily, and the filter is still unused)
   - Choosing a unique identifier of a dynamic filter based on a raw pointer address u64 representation forces you to perform calls to the operating system in serialization code (`std::process::id()`) that would not be necessary with a proper dynamic filter id.
   
   My impression is: the usage of raw pointers addresses as proxy to business-logic details in dynamic filters has gone too far, and we should be using proper ids instead.



##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   I see one of the main issues is how snapshotting works with dynamic filtering.
   
   In the physical expression trait:
   
   ```rust
       ...
       /// A system or function that can only deal with a hardcoded set of `PhysicalExpr` implementations
       /// or needs to serialize this state to bytes may not be able to handle these dynamic references.
       /// In such cases, we should return a simplified version of the `PhysicalExpr` that does not
       /// contain these dynamic references.
       ...
       fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>>;
   ```
   
   This doesn't return a snapshot of the `DynamicFilterPhysicalExpr`, it returns a snapshot to the inner computed physical expression, losing all the information of the original  `DynamicFilterPhysicalExpr`.
   
   What we want in this PR, is that what we serialize is the actual `DynamicFilterPhysicalExpr`, with all it's details, so probably ` snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at all for any expression, and we should be able to treat the dynamic filter as just another serializable `PhysicalExpr` 
   



##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +291,52 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety

Review Comment:
   Safety comments in Rust are typically given for when the function has an `unsafe` block, and the dev needs to explain what are the implicit safety constraints for users to use that function.
   
   As this function is not `unsafe`, I'd suggest just choosing a different word.



##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
     codec: &dyn PhysicalExtensionCodec,
     proto_converter: &dyn PhysicalProtoConverterExtension,
 ) -> Result<protobuf::PhysicalExprNode> {
+    // Check for DynamicFilterPhysicalExpr before snapshotting.
+    // We need to handle it before snapshot_physical_expr because snapshot()
+    // replaces the DynamicFilterPhysicalExpr with its inner expression.
+    if let Some(df) = value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {

Review Comment:
   Doubling down on the above, the fact that dynamic filters cannot be treated as normal expressions makes me think that there might be better ways of dealing with them.
   
   Maybe there's an opportunity for making them normal expressions again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851408492


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   Anyways, still applicable, if dynamic filters where using proper ids, we wouldn't need to resource to things like:
   ```
   if node.downcast<DynamicFilterExpr> {
       // get also a different identifier based on an inner pointer address 
   }
   // get an identifier based on a pointer address
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851394952


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -327,6 +468,14 @@ impl DynamicFilterPhysicalExpr {
         Arc::strong_count(self) > 1 || Arc::strong_count(&self.inner) > 1
     }
 
+    /// Returns a unique identifier for the inner shared state.
+    ///
+    /// Useful for checking if two [`Arc<PhysicalExpr>`] with the same
+    /// underlying [`DynamicFilterPhysicalExpr`] are the same.
+    pub fn inner_id(&self) -> u64 {
+        Arc::as_ptr(&self.inner) as *const () as u64

Review Comment:
   🤔 I see that the `std::process::id()` thing was already there before...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2841152921


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -497,15 +646,23 @@ mod test {
             &filter_schema_1,
         )
         .unwrap();
-        let snap = dynamic_filter_1.snapshot().unwrap().unwrap();
-        insta::assert_snapshot!(format!("{snap:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);
+        let df1 = dynamic_filter_1
+            .as_any()
+            .downcast_ref::<DynamicFilterPhysicalExpr>()
+            .unwrap();
+        let current1 = df1.current().unwrap();
+        insta::assert_snapshot!(format!("{current1:?}"), @r#"BinaryExpr { left: Column { name: "a", index: 0 }, op: Eq, right: Literal { value: Int32(42), field: Field { name: "lit", data_type: Int32 } }, fail_on_overflow: false }"#);

Review Comment:
   Ah thanks. I missed this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2842048151


##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3818,6 +3822,18 @@ impl DeduplicatingSerializer {
             session_id: rand::random(),
         }
     }
+
+    fn hash(&self, ptr: u64) -> u64 {
+        // Hash session_id, pointer address, and process ID together to create expr_id.
+        // - session_id: random per serializer, prevents collisions when merging serializations
+        // - ptr: unique address per Arc within a process
+        // - pid: prevents collisions if serializer is shared across processes
+        let mut hasher = DefaultHasher::new();
+        self.session_id.hash(&mut hasher);
+        ptr.hash(&mut hasher);
+        std::process::id().hash(&mut hasher);
+        hasher.finish()
+    }
 }
 
 impl PhysicalProtoConverterExtension for DeduplicatingSerializer {

Review Comment:
   I thought DeduplicatingSerializer already handled dynamic filters without any further work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] proto: serialize and dedupe dynamic filters [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2956617056


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -180,18 +291,52 @@ impl DynamicFilterPhysicalExpr {
         }
     }
 
+    /// Create a new [`DynamicFilterPhysicalExpr`] from `self`, except it overwrites the
+    /// internal state with the source filter's state.
+    ///
+    /// This is a low-level API intended for use by the proto deserialization layer.
+    ///
+    /// # Safety

Review Comment:
   Removed



##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -3918,24 +3936,51 @@ impl PhysicalProtoConverterExtension for DeduplicatingDeserializer {
     where
         Self: Sized,
     {
-        if let Some(expr_id) = proto.expr_id {
-            // Check cache first
-            if let Some(cached) = self.cache.borrow().get(&expr_id) {
-                return Ok(Arc::clone(cached));
+        // The entire expr is cached, so re-use it.
+        if let Some(expr_id) = proto.expr_id
+            && let Some(cached) = self.cache.borrow().get(&expr_id)
+        {
+            return Ok(Arc::clone(cached));
+        }
+
+        // Cache miss, we must deserialize the expr.
+        let mut expr =
+            parse_physical_expr_with_converter(proto, ctx, input_schema, codec, self)?;
+
+        // Check if we need to share inner state with a cached dynamic filter
+        if let Some(dynamic_filter_id) = proto.dynamic_filter_inner_id {

Review Comment:
   Removed downcasts



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org