You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2025/08/25 19:46:50 UTC
[I] COALESCE expr in datafusion spark crate should perform lazy evaluation of the operands [datafusion]
coderfender opened a new issue, #17322:
URL: https://github.com/apache/datafusion/issues/17322
### Describe the bug
We observed that datafusion-spark's coalesce should perform lazy evaluation of the operands from left to right similar to spark . Current implementation throws an exception in case any operand throws an error which is not in alignment with spark which does lazy evaluation to return first non NULL values
### To Reproduce
`select coalesce(100, 1/0) from tbl`
### Expected behavior
`select coalesce(100, 1/0) from tbl`
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17322: COALESCE expr in datafusion should perform lazy evaluation of the operands
URL: https://github.com/apache/datafusion/issues/17322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3508072510
We merged a fix for this in the 51 release, but we have found as subtle problem so I am planning to revert it for 51 (the fix should appear in 52):
- https://github.com/apache/datafusion/pull/18567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion spark crate should perform lazy evaluation of the operands [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3221532142
cc : @andygrove , @kazuyukitanimura
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion spark crate should perform lazy evaluation of the operands [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3221543369
For reference, this is the behavior in Postgres:
```
postgres=# select coalesce(1, 1/0);
coalesce
----------
1
(1 row)
postgres=# select coalesce(1/0, 1);
ERROR: division by zero
```
This differs from DataFusion's behavior:
```
DataFusion CLI v48.0.0
> select coalesce(1, 1/0);
Arrow error: Divide by zero error
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3868428688
Reviving this thread to update comet side of things once we bump DF version to 52 @comphead , @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3872415426
This query works well now in datafusion 52
```sql
DataFusion CLI v52.1.0
> select coalesce(100, 1/0) from values (1), (2);
+------------------------------------------+
| coalesce(Int64(100),Int64(1) / Int64(0)) |
+------------------------------------------+
| 100 |
| 100 |
+------------------------------------------+
2 row(s) fetched.
Elapsed 0.038 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3221555151
Ah that explains it (fixed the title) . Perhaps we are adhering to postgreSQL 's coalesce implementation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-4007304990
Thank you for the update @alamb . I will revert comet's fix to use DF's coalesce function then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion spark crate should perform lazy evaluation of the operands [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3221551451
@coderfender This issue is for DataFusion's `coalesce` expression and is not specific to `datafusion-spark` as far as I know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] COALESCE expr in datafusion should perform lazy evaluation of the operands [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #17322:
URL: https://github.com/apache/datafusion/issues/17322#issuecomment-3223167455
I can work on this once the comet's side impl is fixed .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org