You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "marc-pydantic (via GitHub)" <gi...@apache.org> on 2025/10/27 13:16:17 UTC

[PR] Do not accept null is set [datafusion]

marc-pydantic opened a new pull request, #18301:
URL: https://github.com/apache/datafusion/pull/18301

   ## Which issue does this PR close?
   
   - Closes #18300
   
   ## Rationale for this change
   
   As laid out in the issue, this improves internal checks by testing an assumed invariant, instead of silently nulling data on error. The cost is a single null check on a column with a number of entries dependent on the number of partitions, not the data itself.
   
   ## What changes are included in this PR?
   
   * Adds a null check to the second column of `merge_batch` of both `FIRST_VALUE` and `LAST_VALUE`.
   
   ## Are these changes tested?
   
   Tests are included.
   
   ## Are there any user-facing changes?
   
   Hopefully not.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "marc-pydantic (via GitHub)" <gi...@apache.org>.
marc-pydantic commented on PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#issuecomment-3451899841

   Ah, I wish I had seen that before I turned it into asserts... Anyway, I can roll back 3 commits to use internal errors, or use assertions, your choice :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#discussion_r2467645023


##########
datafusion/functions-aggregate/src/first_last.rs:
##########
@@ -810,6 +810,11 @@ impl Accumulator for TrivialFirstValueAccumulator {
         // Second index contains is_set flag.
         if !self.is_set {
             let flags = states[1].as_boolean();
+            debug_assert!(

Review Comment:
   Sorry for the back and forth! 😅 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is_set for first_value/last_value [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #18301:
URL: https://github.com/apache/datafusion/pull/18301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "marc-pydantic (via GitHub)" <gi...@apache.org>.
marc-pydantic commented on PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#issuecomment-3451584965

   I'm totally fine with this being a `debug_assert`, just was reluctant to add one. I'll update accordingly!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "marc-pydantic (via GitHub)" <gi...@apache.org>.
marc-pydantic commented on PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#issuecomment-3456837761

   To be fair, I [personally](https://compilersaysno.com/posts/the-elements-of-rust-error-handling/) would also prefer internal errors, I was just deferring to you guys here. The problem occurring is an indicator of an input validation failure, while ideally this should have happened earlier and be encoded in the type system, this is probably the next best place to put this.
   
   If there's an assert, it'd be an API contract violated, so for completeness sake, I assume we'd have to put additional validation code that looks similar to the internal error one outside the function.
   
   As for the computational overhead, if I am not mistaken in reading the `arrow-buffer` source, this will be handled as counting bits on a packed bitvector, thus count ones will be used. Assuming some moderate compilation, I'd say the overhead for all cases of <=64 entries --- which should be the vast majority of cases? --- is a single CPU instruction, plus one branch not taken. I'd consider that negligible, but will happily be convinced otherwise!
   
   I've reverted the respective commits that turned it into macros, as requested :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#discussion_r2465672445


##########
datafusion/functions-aggregate/src/first_last.rs:
##########
@@ -1473,6 +1481,20 @@ impl Accumulator for LastValueAccumulator {
     }
 }
 
+/// Validates that is_set flags do not contain NULL values.
+fn validate_is_set_flags(flags: &BooleanArray, function_name: &str) -> Result<()> {
+    if flags
+        .nulls()
+        .map(|nulls| nulls.null_count() > 0)
+        .unwrap_or_default()
+    {

Review Comment:
   ```suggestion
       if flags.null_count() > 0 {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is_set for first_value/last_value [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#issuecomment-3457029075

   Thanks @marc-pydantic, sorry for causing some back and forth 😅 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#discussion_r2466354252


##########
datafusion/functions-aggregate/src/first_last.rs:
##########
@@ -810,6 +810,11 @@ impl Accumulator for TrivialFirstValueAccumulator {
         // Second index contains is_set flag.
         if !self.is_set {
             let flags = states[1].as_boolean();
+            debug_assert!(

Review Comment:
   I agree with @Jefffrey  it would be nice to have this be an internal error rather than an assert (which results in a panic in debug build and are silently ignored on release builds)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Do not accept null is set [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #18301:
URL: https://github.com/apache/datafusion/pull/18301#issuecomment-3451616697

   > I'm totally fine with this being a `debug_assert`, just was reluctant to add one. I'll update accordingly!
   
   tbh Internal error is fine as well i guess, since thats what Internal error enum variant is supposed to be for 😅 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org