You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2026/03/29 10:39:44 UTC
[I] CaseWhen does not work with custom implemented column expression [datafusion]
rluvaton opened a new issue, #21231:
URL: https://github.com/apache/datafusion/issues/21231
### Describe the bug
While upgrading to DataFusion 52 (I know, I'm getting to 53 slowly) tests started failing for CASE WHEN for us because we have cases where we use custom `Column` `PhysicalExpr` rather than DataFusion one.
this should not affect correctness.
This was done as an optimization in this PR by @pepijnve
- #18329
because it find the columns referenced by traversing the expression tree and finding the `Column` Physical Expression. however, when having custom `Column` impl it will not add that used column to the projection
### To Reproduce
This is the same test as:
https://github.com/apache/datafusion/blob/1a0af76f780708ff9d9e1940686748bd743352d3/datafusion/physical-expr/src/expressions/case.rs#L1694-L1732
but using Custom impl of `Column`
```rust
#[test]
fn case_without_expr_and_with_custom_column_impl() -> Result<()> {
/// Custom impl of column
#[derive(Debug, Hash, PartialEq, Eq, Clone)]
pub struct CustomColumn {
/// The name of the column (used for debugging and display purposes)
name: String,
/// The index of the column in its schema
index: usize,
data_type: DataType,
nullable: bool,
}
impl CustomColumn {
pub fn new_with_schema(name: &str, schema: &Schema) -> Result<Arc<dyn PhysicalExpr>> {
let index = schema.index_of(name)?;
let field = schema.field(index);
Ok(Arc::new(CustomColumn {
name: name.to_string(),
index,
data_type: field.data_type().clone(),
nullable: field.is_nullable(),
}))
}
}
impl std::fmt::Display for CustomColumn {
fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
write!(f, "{}@{}", self.name, self.index)
}
}
impl PhysicalExpr for CustomColumn {
fn as_any(&self) -> &dyn Any {
self
}
fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
Ok(self.data_type.clone())
}
fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
Ok(self.nullable)
}
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
self.bounds_check(batch.schema().as_ref())?;
Ok(ColumnarValue::Array(Arc::clone(batch.column(self.index))))
}
fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
vec![]
}
fn with_new_children(
self: Arc<Self>,
_children: Vec<Arc<dyn PhysicalExpr>>,
) -> Result<Arc<dyn PhysicalExpr>> {
Ok(self)
}
fn fmt_sql(&self, _: &mut Formatter<'_>) -> std::fmt::Result {
unimplemented!()
}
}
impl CustomColumn {
fn bounds_check(&self, input_schema: &Schema) -> Result<()> {
if self.index < input_schema.fields.len() {
Ok(())
} else {
internal_err!(
"PhysicalExpr CustomColumn references column '{}' at index {} (zero-based) but input schema only has {} columns: {:?}",
self.name,
self.index,
input_schema.fields.len(),
input_schema.fields().iter().map(|f| f.name()).collect::<Vec<_>>()
)
}
}
}
let batch = case_test_batch()?;
let schema = batch.schema();
// CASE WHEN a = 'foo' THEN 123 WHEN a = 'bar' THEN 456 END
let when1 = binary(
CustomColumn::new_with_schema("a", &schema)?,
Operator::Eq,
lit("foo"),
&batch.schema(),
)?;
let then1 = lit(123i32);
let when2 = binary(
CustomColumn::new_with_schema("a", &schema)?,
Operator::Eq,
lit("bar"),
&batch.schema(),
)?;
let then2 = lit(456i32);
let expr = generate_case_when_with_type_coercion(
None,
vec![(when1, then1), (when2, then2)],
None,
schema.as_ref(),
)?;
let result = expr
.evaluate(&batch)?
.into_array(batch.num_rows())
.expect("Failed to convert to array");
let result = as_int32_array(&result)?;
let expected = &Int32Array::from(vec![Some(123), None, None, Some(456)]);
assert_eq!(expected, result);
Ok(())
}
```
### Expected behavior
to work with any custom column impl, and hopefully still having that optimization
### Additional context
also might affect lambda support:
- https://github.com/apache/datafusion/pull/18921#discussion_r3006009469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4150928246
In my case, lambda variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4378523250
@gstvg I think the approach where the indices are kept stable in the public physical plan and the rewriting is an internal optimisation is very sensible. Very easy to reason about.
This approach also aligns perfectly with the collect/project idea I sketched above. Would we still want to introduce methods on `PhsyicalExpr` to support this? We would essentially be replacing the current downcasting code with polymorphic method calls which could support custom column expressions provide we accept the 'extend/append only' model.
The change you made in the lambda column capture branch where `IndexSet` was replaced with `BTreeSet` is an interesting one from an API design pov and is why I was asking questions earlier about how to collect the column indices in a recursive method call. I'm a bit reluctant to make the output collection type a part of the signature since that's up to the caller to decide. One caller may want to count occurrences of each index, one may not care about order while another does, etc. That kind of nudges in the direction of a dyn function callback rather than a mutable reference to some collection. Is there an idiomatic pattern in Rust for something this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4164600743
> A lambda function defines a new naming scope. [..] I’m also thinking something along the lines of a tree traversal where the schema can change in the downward phase, and indices can be remapped in the upward phase. You would need to have the schema available for this though, which is not the case when the projection logic is currently triggered.
what I'm saying is that for you should stay in the same scope as the case expression, if some child expression create a new scope, you should not get into that scope. and therefor should not have the schema available
all you need is for expression that create a new scope to not return their children
so for `array_transform(<expr>, value -> <expr-2>)`, array_transform expression will only return <expr-1> and not the lambda, since the lambda is a new scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4373345570
Column index binding happens much earlier than the case projection though, doesn't it? I'll check the outer column reference PR to see what's being proposed there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4150087182
Just for my understanding, could you explain the purpose of the custom column reference implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4149886698
When implementing the same optimization for lambda expression, I choose to replace unreferenced columns with a `NullArray` as it's `filter/take` is almost a no-op. I guess the same can done here, and also simplify things(no need to keep both the unprojected as well as the projected expression)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4343526798
> Thus I think the "right" way to fix this bug (and avoid potential future bugs) is:
> 1. Add an API to PhysicalExpr to report schema column used
@alamb I agree with you that we should try to make the aspect of 'which columns do you need' and 'project yourself with this column mapping' part of the `PhysicalExpr` trait so that each implementation can handle this however it sees fit. That allows us to remove the downcasts from `Case` and wherever else similar logic exists.
The part that I'm still not confident about is how we would control recursion. Would it be preferable for the 'collect columns' trait method to return the used columns for the physical expression itself and all its children (in other words, the method implementation itself handles the recursion) or should the recursion be done externally using a visitor?
The tradeoff I see is that internal recursion potentially requires a lot of data copying. If the method returns a `Vec` for instance you have to merge Vecs at each recursion level, filter out duplicates, etc.
With external recursion though we can avoid that and collect into a set, but I think we may need to use something like `Result<(Option<usize>, TreeNodeRecursion)>` as return type so that implementations can return an optional column index and also have scope control.
A possible hybrid is internal recursion, but rather than returning values via the return value, we do so by passing in a consumer callback function. That avoids having to have `TreeNodeRecursion` in the method signature, and also avoids the need for recursive result merging, but then we probably have to have a dyn function parameter.
Is there any prior art in the library for something like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4164883881
no, and I already have it in my repo.
because lets say I have this:
```
select external_column, array_transform(list, item -> item + external_column) from tbl
```
so array_transform knows that it has 1 column that it should get from the parent scope
then this is what the implementation will save:
```
ArrayTransform:
child: column list
external references mapping:
// (name used in the lambda expression, reference expression)
"external_column", column(external column)
lambda:
Add:
bound lambda variable: position: 0
bound lambda variable: position 1
```
and when you create the schema in the lambda expression you map from external used references to the internal column position
and when case expression rewrite the columns, it rewrite the external reference mapping
Look at this example implementation of array transform with support for external reference capture that will work with the mapping
```rust
struct ArrayTransformExpression {
child: Arc<dyn PhysicalExpr>,
lambda_function: Arc<dyn PhysicalExpr>,
external_references: Vec<(
// Renamed schema field name in lambda schema
String,
// the expression to bound
Arc<dyn PhysicalExpr>
)>
}
impl ArrayTransformExpression {
fn create_lambda_function_schema(&self, input_schema: &Schema, lambda_args: Vec<FieldRef>) -> SchemaRef {
let mut lambda_schema_fields = lambda_args.clone();
let start_index = lambda_schema_fields.len()
for (field_name, external_ref) in self.external_references {
let field = external_ref.return_field(input_schema).unwrap();
let renamed_field = field.with_name(field_name);
lambda_schema_fields.push(renamed_field);
}
Arc::new(Schema::new(lambda_schema_fields))
}
fn create_lambda_function_input_batch(&self, input_batch: &RecordBatch, lambda_args: Vec<(FieldRef, ArrayRef)>) -> SchemaRef {
let (mut lambda_schema_fields, mut lambda_arrays) = lambda_args.into_iter().unzip();
let start_index = lambda_schema_fields.len()
for (field_name, external_ref) in self.external_references {
let field = external_ref.return_field(input_schema).unwrap();
let renamed_field = field.with_name(field_name);
lambda_schema_fields.push(renamed_field);
let external_reference_value = external_ref.evaluate(input_batch);
lambda_arrays.push(external_reference_value);
}
let lambda_function_schema = Arc::new(Schema::new(lambda_schema_fields));
RecordBatch::try_new(lambda_function_schema, lambda_arrays)
}
fn children_in_scope(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
std::iter::once(&self.child).chain(self.external_references.iter().map(|(_, external_ref)| external_ref)).collect()
}
fn with_new_children_in_scope(
self: Arc<Self>,
children_in_scope: Vec<Arc<dyn PhysicalExpr>>,
) -> Result<Arc<dyn PhysicalExpr>> {
// child and external references
assert_eq!(children_in_scope.len(), 1 + self.external_references.len())
Ok(Arc::new(ArrayTransformExpression {
child: children_in_scope[0],
lambda_function: self.lambda_function.clone(),
// this update the mapping
external_references: self.external_references
.iter()
.map(|(name, _old_ref)| name)
.zip(children_in_scope.iter().skip(1))
.collect()
}))
}
}
```
so in your case, if you work through children_in_scope and keep the same code you already do in CASE WHEN it will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4177839220
From what I can tell, the core issue is that CaseWhen currently assumes input column references can be discovered by finding built-in `Column` physical exprs. That is not true for @rluvaton’s system, and lambda scope makes it even trickier.
I think there are two possible solutions:
# Clean solution
The cleanest fix seems to be adding an API to [PhysicalExpr](https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html) to report in-scope input column usage, along the lines of what @pepijnve proposed in [this comment](https://github.com/apache/datafusion/issues/21231#issuecomment-4151111735) (column_index() / required_column_indices()).
That is probably a non-trivial change, since I believe there are several places in DataFusion that downcast to `Column`.
# Possible workaround
One thing that might work for @rluvaton 's case is to make his `CustomExpr` is to report a real `Column` child for any expression that refers to the outer schema
```rust
struct MyCustomColumn {
// if this expression refers to a outer schema column (rather than something
// bound in the lambda
outer_column: Option<Arc<dyn PhysicalExpr>>,
}
impl PhysicalExpr MyCustomColumn
..
fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
// if the expression in the Lambda refers to an outer schema column, return
// a reference to that column here:
if let Some(outer_column) = self.outer_column {
vec![outer_column]
} else {
vec![];
}
}
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
// if this is just a wrapper, use the inenr column
if let Some(outer_column) = self.outer_column {
outer_column.evaluate(batch)
} else {
// otherwise, do whatever is required for columns that are bound in lambda
...
}
}
}
```
That way when traversing via the [PhysicalExpr::children](https://docs.rs/datafusion/latest/datafusion/physical_expr/trait.PhysicalExpr.html#tymethod.children) the rest of DataFusion would still see a `Column` expr for actual (outer) schema references
This would require some sort of analysis pass on the lambda to identify the outer column `Column` references, but I suspect that already exists somewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4152648709
```
case x@0
when y@1 then array_transform(z@2, l@3 -> l@3 + 1)
end
```
That's how it will look when column capture support get's reintroduced.
Before projecting, can't we just remove the out of bounds indexes relative to the batch being projected? Since lambda variables are always pushed to the end of the batch, all of the out of bounds ones should refer lambda variables not yet available.
```rust
fn case_when_with_expr(
&self,
batch: &RecordBatch,
projected: &ProjectedCaseBody,
) -> Result<ColumnarValue> {
let return_type = self.data_type(&batch.schema())?;
// projected.projection may include indexes of lambda variables not available on this batch
let projection = projected
.projection
.iter()
.copied()
.filter(|index| *index < batch.num_columns())
.collect::<Vec<_>>();
if projection.len() < batch.num_columns() {
let projected_batch = batch.project(&projection)?;
projected
.body
.case_when_with_expr(&projected_batch, &return_type)
} else {
self.body.case_when_with_expr(batch, &return_type)
}
}
```
There are cases where variables should be projected:
```
array_transform(
x@0,
a@2 -> case
when y@1 > 2 then array_transform(a@2, b@3 -> b@3 +1)
else then array_transform(a@2, c@3 -> c@3 * 2)
end
)
```
Here, CaseWhen should project `a@2` but not `b@3` nor `c@3`
To be stricter, we can also modify the column indices collecting tree traversal to keep track of new lambdas variables introduced by lambda nodes, and skip projecting them. On the example above, CaseWhen can see the lambdas `b -> b + 1` and `c -> c*2`, store the name of it's parameters `b` and `c`, and skip collecting the indices of any variable with those names. Since it wouldn't have seen the lambda `a -> case ..`, it should project the index of variable named `a` normally
```
case x@0
when y@1 then array_transform(z@2, l@0' -> l@0' + 1)
end
```
That's how it looks like today at https://github.com/apache/datafusion/pull/18921/changes/1b7f4bfd2c3aa9d9e3c431775e199dc4113e27d3. Since there's no column capture for now and no regular column can exist within a lambda body, we just jump all lambda nodes, and none of the variables of it's body will be visited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4389149703
> This approach also aligns perfectly with the collect/project idea I sketched above. Would we still want to introduce methods on PhsyicalExpr to support this? We would essentially be replacing the current downcasting code with polymorphic method calls which could support custom column expressions provide we accept the 'extend/append only' model.
In my opinion, the general idea of switching to methods on PhysicalExpr rather than downcasting is a good idea in general as it makes the code easier to reason about and the contracts of what PhysicalExprs do / are used for more explicit in the trait (rather than having to look through the codebase at the uses)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4377092569
At #21323, during physical planning, lambda scope schemas are created by *appending* the lambda parameters to the outer schema, so that the outer schema fields still exists in the same position, regardless of being referenced or not
(how it handle name shadowing I believe it's not relevant for this discussion). Given this query:
```sql
create table t as select [[1, 2]] as b, 2 as b;
select array_transform(a, arr -> array_transform(arr, v -> v + b)) from t;
```
The schemas would be:
```text
[0 => a, 1 => b]
[0 => a, 1 => b, 2 => arr]
[0 => a, 1 => b, 2 => arr, 3 => v]
```
The planned query with indices, which would be what is visible via tree node traversals:
```sql
select array_transform(a@0, arr@2 -> array_transform(arr@2, v@3 -> v@3 + b@1)) from t;
```
So every scoped schema is a superset of the root schema, and every column reference, regardless of being
within a lambda scope, has an index that it's valid relative to the root schema (lambda variable references uses a different expression type)
Then, as an internal optimization, not visible to the external world, similar to the case optimization,
lambda derives a projected body to work with projected batches that doesn't include unrefereced columns or lambda variables:
```text
[0 => a, 1 => b]
[0 => b, 1 => arr]
[0 => b, 1 => v]
```
```sql
select array_transform(a@0, arr@1 -> array_transform(arr@1, v@1 -> v@1 + b@0)) from t;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4347666107
I don't quite follow the options you are proposing @pepijnve
You could implement internal recursion by passing in the result Vec (e.g. `output: &mut Vec<...>`) but I am not sure what is what you mean
I don't think an API that reports "what columns do you need" would handle "scoped" access as was originally described above -- as every time the recursion entered a new scope you would have to rebind the column definitions somehow
However I am not sure I am answering the questions you had in mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4152964279
@gstvg yes, when the column extraction logic is “lambda aware” we can certainly implement this correctly. What Inwas trying to think about is how we can write it so that third-party code that introduces new constructs does not get broken. In other words, what extension points do we need to allow @rluvaton’s “custom column” implementation (which I’m using as a proxy for whatever kind of extension people might have built) to work correctly without the DataFusion crate being aware of its existence and semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4188994944
I implemented a change that will allow me to mark it with different scope in:
- #21386
This implementation add `children_in_scope` and `with_new_children_in_scope` with relevant scoped tree traversal.
again this does not solve the original issue for user provided column but it solve the root cause that I used custom column in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4158047284
What you’re describing is exactly the bit I’m wondering how to solve. A lambda function defines a new naming scope. Which is quite different from any other PhysicalExpr. If we need to assume that any PhysicalExpr can introduce a new concept like lambdas, then what kind of api do we need to be able to deal with that? I’m also thinking something along the lines of a tree traversal where the schema can change in the downward phase, and indices can be remapped in the upward phase. You would need to have the schema available for this though, which is not the case when the projection logic is currently triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4164613280
Wouldn’t that make it impossible to implement column references from within lambdas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4374127249
> Column index binding happens much earlier than the case projection though, doesn't it? I'll check the outer column reference PR to see what's being proposed there.
I guess I was thinking that physical planning would do whatever column binding was needed 🤔
then any reference to a non base column would be via some other type of physical exper (not Column)
Then the implementation of lambda would pass in the appropriate input somehow
Given @gstvg merged something for lambdas recently maybe they already sorted this out. I am not sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4389155795
> hat kind of nudges in the direction of a dyn function callback rather than a mutable reference to some collection. Is there an idiomatic pattern in Rust for something this?
What about a visitor pattern (instead of hard coded APIs)? Maybe the existing treenode api can help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4337811288
I know, I was just wondering what examples of custom columns are. The example you gave was lambda variables, but those now exist in the core library. Are there any other examples of custom columns that would make this useful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4337288874
the underlying problem is still the same, custom column are not supported while before they were
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4337127784
@rluvaton with the initial lambda support landed on main, I'm wondering if we should consider punting on the general solution and just ensure the projection logic in case handles lambdas correctly (if that's not already the case). Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4178732253
Since lambda variables are also evaluated via the batch, `CaseWhen` projection must see them as well. `MyCustomColumn` could return unbound columns so `CaseWhen` could see them, but others traversals that aren't lambda aware would see them too and would off course break.
@rluvaton `children_in_scope/with_new_children_in_scope` does work, but I believe it would actually require new `TreeNode` methods, `apply_in_scope` for `CaseWhen`, but others should be added as well. And if the regular `children/with_new_children` don't report the `external_references`, existing traversals that look for `Column`s would need use them instead, and also I believe there would be no way to do a traversal that contains both the bound `Column`s as well lambda bodies, only one with columns but without lambdas (`_in_scope` versions) or another with lambdas but without columns (existing ones).
When lambda support arrives with `CaseWhen` being aware of it, and @rluvaton can use it instead of it's custom implementation (which I believe will happen since he's helping me implement it on #18921), then this became a non issue, right? I don't think there will be any other case of custom column-like expressions besides regular columns and lambda variables. And then, If someone needs either a custom `Column` or `LambdaVariable` (but not something else), then it can return the core version as children as proposed by @alamb
I have implemented lambda column capture support compatible with `CaseWhen` including the same optimization as `CaseWhen` at #21323 ([tests](https://github.com/apache/datafusion/pull/21323/changes/5c5ca195d1fc2c708a1e4964e8392337fc3b0b02#diff-82f03654507864698a026045144642b3dcda8aedc20d29aea782f28b47b9d0ea)), without requiring any major or public facing changes. Every `Column` present on lambda body exposed via `children` refers to correct index of the outermost, "root" schema.
As a last resort, we can evaluate `LambdaVariable` without the batch by adding a `Option<ArrayRef>` to it that must be set before evaluation:
```rust
struct LambdaVariable {
name: String,
field: FieldRef,
value: Option<ArrayRef>,
}
```
It would require a tree rewrite for every batch evaluation (internally handled by `HigherOrderFunctionExpr`) but would free others parts of the code to care about lambdas at all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4194002489
I think the root problem is that currently there is an implicit invariant that for any `PhysicalExpr` all of its children are evaluated against the same `Schema`
the Case logic now relies on this implicit invariant which is why you are hitting an error
While you can modify case as is proposed in https://github.com/apache/datafusion/pull/21386 that will not protect you from other parts of the DataFusion code relying on the same assumption (either existing or that will be added in the future)
Thus I think the "right" way to fix this bug (and avoid potential future bugs) is:
1. Add an API to PhysicalExpr to report `schema column used`
2. Update case to use that API
3. Update your lambda expression implementation so that a `Column` refers to a base table, and rewrite `Column`s that refer to columns created by the lambda, use a different physical expression for them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4157922228
the case when that I have a problem with is inside the lambda function, like
```
transform(list, x => when x > col('other_col') then col('other_col') when x < col('another_col') then x else -2)
```
------
> Before projecting, can't we just remove the out of bounds indexes relative to the batch being projected? Since lambda variables are always pushed to the end of the batch, all of the out of bounds ones should refer lambda variables not yet available.
No, because of the index is inbound? you still don't want to capture that
------
the problem beside the API for detecting and creating with different index custom column implementation is that you should traverse the expression through `children` function, as it can go too deep
Lets say we have this expression:
```
array_transform(
very_nested_list,
nested_list =>
case
when something
then transform(
concat(nested_list, external_array),
list => len(list) + len(nested_list)
)
)
```
if you get all expression that implement `column_index` so in the subtree under the CASE WHEN you will get the following expressions:
```
nested_list, external_array, list, nested_list
```
the `list` here is not bounded to the same schema as the `CASE WHEN` context, it is in a different schema, so the referenced index might not exist in the schema that the `CASE WHEN` is in.
so you should **not** traverses/modify the tree through `children` as it will lead you to that. instead you should traverse/modify through children that have the same input schema as you, and then the innermost `transform` lambda physical expression will not return expression under the lambda function itself, and only under `concat(nested_list, external_array)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4163888780
> you only need to pass throught the same naming scope
That's a bit too cryptic for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4175666286
Ok, I think I see where you're going. I think I might have been thrown off a bit by the term "custom column reference".
In your example, you still have to use the builtin `Column` implementation to reference a column. Is it acceptable that we have an API that has that expectation, or would we want to abstract away the usage of `Column` as well?
The API you're proposing is in terms of naming scope, the alternative would be in terms of projection where we can extract column indices and rewrite using a projection array. If we leave the recursion up to each `PhysicalExpr` we can achieve the same effect that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4188619900
`children` must return all expressions that the expression is using, otherwise how would you check if a given expression exists anywhere in the tree or do any kind of traversal that doesn't care about the schema, like wrap every expression with debug expression that print the input and output, or add timing and so on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4389139761
> So every scoped schema is a superset of the root schema, and every column reference, regardless of being
within a lambda scope, has an index that it's valid relative to the root schema (lambda variable references uses a different expression type)
This is very clever. Nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4372693252
I think a classic way to implement this type of resolution is using a stack of named scope / bindings.
So that would be something like
```
(Query Bindings (column references)
-> Local Scope Bindings (any newly defined references)
...
-> nested Scope Bindings for a nested lambda, for exampe (any newly defined references)
```
So then when we went to resolve a mapping `"a"` to a column index it would look back in the stack of scope bindings.
When the recursion passes a lambda function, it would push a new scope on ot the stack and pop off the stack when done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4151111735
My first thought was to add something like this
```
pub trait PhysicalExpr {
fn column_index(&self) -> Option<usize> {
None
}
}
```
to avoid the downcast to `Column`.
For something like this (not real syntax, just making some educated guesses based on what you said)
```
case x@0
when y@1 then array_transform(z@2, l@3 -> l@3 + 1)
end
```
where the lambda variable is an extra index compared to the parent scope, or
```
case x@0
when y@1 then array_transform(z@2, l@0' -> l@0' + 1)
end
```
where `0'` is a redefined index that's not going to be sufficient though. Without knowledge of the existence of lambda functions you can't really do the right thing. In the first alternative you would end up trying to project a column index that doesn't exist from the perspective of the record batch coming into the case expression. In the second alternative you would end up retaining an existing column index, but that's actually a misinterpretation of the index.
One way around that might be to push the recursion into `PhysicalExpr` with something like
```
fn required_column_indices(&self) -> Vec<usize> {
self.children()
.iter()
.map(|child| child.required_column_indices())
.flatten()
.sorted()
.unique()
.collect()
}
```
but I don't think that's going to get us to a solution for lambda functions either. I only just started looking at the lambda PR. I'll continue with that tomorrow; maybe I can find some inspiration there.
Short term, we might just have to disable the 'project' code path. It could be turned into an optimisation instead with a dedicated 'project' wrapper expression. That way you could at least opt-out if it's getting in the way (or alternatively make it opt-in only).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4160717417
you only need to pass throught the same naming scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4156064225
I think something similar to `children` and `with_new_children`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4188630560
> [@rluvaton](https://github.com/rluvaton) `children_in_scope/with_new_children_in_scope` does work, but I believe it would actually require new `TreeNode` methods, `apply_in_scope` for `CaseWhen`, but others should be added as well.
yes and the default implementation for `children_in_scope/with_new_children_in_scope` would either be to call `with_children` to avoid any breaking changes.
> When lambda support arrives with `CaseWhen` being aware of it, and [@rluvaton](https://github.com/rluvaton) can use it instead of it's custom implementation (which I believe will happen since he's helping me implement it on [#18921](https://github.com/apache/datafusion/pull/18921)), then this became a non issue, right? I don't think there will be any other case of custom column-like expressions besides regular columns and lambda variables. And then, If someone needs either a custom `Column` or `LambdaVariable` (but not something else), then it can return the core version as children as proposed by [@alamb](https://github.com/alamb)
not returning all the children is not a solution in my opinion due to the reasons I listed above, and having custom column implementation should not fail execution even when having lambda expression support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4350321900
Sorry, I didn't provide enough context. I scribbled up my current understanding so I have something to point to.
<img width="3471" height="4275" alt="Image" src="https://github.com/user-attachments/assets/6dc49d3d-1bb7-4cf4-b539-ac6336c7204f" />
I'm using high-order function calls here as an example, but I think this can be generalised to other things as well. If I understood it correctly, the problem we want to handle is the fact that the lambda function in this diagram will use a local schema that is different from the outer schema. The outer one is `a@0, b@1, c@2` while the lambda one is `el@0, c@1`.
My assumption is that somewhere at some point in the code there has to be logic that implements the mapping of the outer `c@2` to the inner `c@1`. In my sketch, my assumption is that that's the 'apply higher order function' node and that is has a mapping table with an entry `2 -> 1`.
A recursive 'collect column indices' function should be able to handle this provided that each node can process the answer from its children. The high-order function application expression node needs to know which of its children are plain expressions, which are functions, what the indices for the lambda variables are, etc. so I would expect it to be able to handle the mapping of the `[0, 1]` answer from the function expression to `[2]` for the outer scope.
As I wrote this, I think I've answered my own question. This is probably easiest to implement with internal recursion (i.e. the trait method provides the answer for the target expression itself and all its children). The alternative would require an extra method to handle the remapping. The other questions I had were more coding style questions regarding how you express output parameters in Rust: literally a `mut` output parameter, a callback, something else?
The more important question though is if the model sketched above is a sufficient solution? Am I overlooking something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CaseWhen does not work with custom implemented column expression [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on issue #21231:
URL: https://github.com/apache/datafusion/issues/21231#issuecomment-4395157327
> What about a visitor pattern (instead of hard coded APIs)? Maybe the existing treenode api can help
That's an option, but that would then remain an external traversal that requires knowledge of all possible PhysicalExpr implementations it needs to handle. If we want to this to be independent of specific implementations we need extra methods on the trait. I'm still a bit on the fence myself if this is a sensible thing to do at this point. Today there are only `Column` and `LambdaVariable` that we would need to support. With just two implementations that are both standard types providing a new extension point may be a bit premature. I'm having a hard time coming with other examples where this might be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org