[GitHub] [arrow-datafusion] crepererum opened a new issue, #3968: `Expr::to_bytes` can produce output that hits `Expr::from_bytes` recursion limit

[GitBox <gi...@apache.org>]

crepererum opened a new issue, #3968:
URL: https://github.com/apache/arrow-datafusion/issues/3968

   **Describe the bug**
   `Expr::to_bytes` with deeply nested expressions can produce a byte sequence that cannot be read by `Expr::from_bytes` because the prost recursion limit is hit. This is suboptimal in distributed application because it would be good if a client can decide if an expression is wire/roundtrip-safe and potentially adjust its predicate pushdown accordingly.
   
   **To Reproduce**
   
   ```rust
   #[test]
   fn roundtrip_deeply_nested() {
   // we need more stack space so this doesn't overflow in dev builds
   std::thread::Builder::new().stack_size(10_000_000).spawn(|| {
   	// don't know what "too much" is, so let's slowly try to increase complexity
   	let n_max = 100;
   
   	for n in 1..n_max {
   	println!("testing: {n}");
   
   	let expr_base = col("a").lt(lit(5i32));
   	let expr = (0..n).fold(expr_base.clone(), |expr, _| expr.and(expr_base.clone()));
   
   	// Convert it to an opaque form
   	let bytes = match expr.to_bytes() {
   		Ok(bytes) => bytes,
   		Err(_) => {
   		// found expression that is too deeply nested
   		return;
   		}
   	};
   
   	// Decode bytes from somewhere (over network, etc.
   	let decoded_expr = Expr::from_bytes(&bytes).expect("serialization worked, so deserialization should work as well");
   	assert_eq!(expr, decoded_expr);
   	}
   
   	panic!("did not find a 'too deeply nested' expression, tested up to a depth of {n_max}")
   }).expect("spawning thread").join().expect("joining thread");
   }
   ```
   
   **Expected behavior**
   Above test passes.
   
   **Additional context**
   Above test fails with:
   
   `thread '<unnamed>' panicked at 'serialization worked, so deserialization should work as well: Plan("Error decoding expr as protobuf: failed to decode Protobuf message: LogicalExprNode.expr_type: BinaryExprNode.l: LogicalExprNode.expr_type: .....recursion limit reached")'`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb closed issue #3968: `Expr::to_bytes` can produce output that hits `Expr::from_bytes` recursion limit

[GitBox <gi...@apache.org>]

alamb closed issue #3968: `Expr::to_bytes` can produce output that hits `Expr::from_bytes` recursion limit
URL: https://github.com/apache/arrow-datafusion/issues/3968


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org