You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "shivbhatia10 (via GitHub)" <gi...@apache.org> on 2025/11/19 15:25:46 UTC
[I] AsyncScalarUDFs break when batch_size doesn't divide number of rows [datafusion]
shivbhatia10 opened a new issue, #18822:
URL: https://github.com/apache/datafusion/issues/18822
### Describe the bug
When `ideal_batch_size` is set to a value that doesn't evenly divide the total number of rows in some data for an `AsyncScalarUDFImpl`, we get this error:
`Error: Internal("Arguments has mixed length. Expected length: 2, found length: 1")`
for example, the numbers vary depending on both parameters.
### To Reproduce
I wrote a test in the fix PR which fails today: https://github.com/apache/datafusion/pull/18819/files
### Expected behavior
Batch size shouldn't necessarily need to divide the number of rows evenly.
### Additional context
I have a fix [PR](https://github.com/apache/datafusion/pull/18819), the issue was in the `AsyncFuncExpr` physical expression. We were calling `ColumnarValue::values_to_arrays` on all the batches returned asynchronously, but this method enforces that all batches need to be the same length, which isn't necessary here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] AsyncScalarUDFs break when batch_size doesn't divide number of rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #18822: AsyncScalarUDFs break when batch_size doesn't divide number of rows
URL: https://github.com/apache/datafusion/issues/18822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org