You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "shehabgamin (via GitHub)" <gi...@apache.org> on 2025/01/22 10:36:50 UTC

[I] DataFusion Regression (Starting in v43): Coercing Various Scalar UDF Arguments to String [datafusion]

shehabgamin opened a new issue, #14230:
URL: https://github.com/apache/datafusion/issues/14230

   ### Describe the bug
   
   A regression was introduced in DataFusion 43.0.0 that impacts the casting of arguments to UTF8 for various Scalar UDFs. Sail's tests uncovered several of these regressions, which required explicit casting in multiple areas as a workaround during the upgrade to DataFusion 43.0.0.
   
   The regressions identified by Sail's tests include the following functions:
   1. `ascii`
   2. `bit_length`
   3. `contains`
   4. `ends_with`
   5. `starts_with`
   6. `octet_length`
   
   Scope of Work:
   - Address the regressions.
   - Port relevant tests from Sail to cover these issues.
   
   ### To Reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   https://github.com/apache/datafusion/issues/14008#issuecomment-2599492511


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Coercing Various Scalar UDF Arguments to String [datafusion]

Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2606874054

   Take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2631350057

   BTW I think @Omega359  hit the same issue about automatically coercing from `int` to `utf8` when testing 45.0.0: https://github.com/apache/datafusion/issues/14008#issuecomment-2629085175
   
   > utf8view, i32 comparison no longer worked
   
   .with_column(
               STRING_FIELD,
               when(col(STRING_FIELD).eq(lit(83)), lit(82)).otherwise(col(STRING_FIELD))?,
           )?
   switching to the obvious change below worked
   
   .with_column(
               STRING_FIELD,
               when(col(STRING_FIELD).eq(lit("83")), lit("82")).otherwise(col(STRING_FIELD))?,
           )?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #14230: DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs
URL: https://github.com/apache/datafusion/issues/14230


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (Int --> String) [datafusion]

Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2632348079

   @alamb @Omega359 It's not only int -> string, pretty much all coercion between types no longer works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2637033235

   I will try and review it shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2667238810

   Close by #14440 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (Int --> String) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2634612836

   > [@alamb](https://github.com/alamb) [@Omega359](https://github.com/Omega359) It's not only int -> string, pretty much all coercion between types no longer works.
   
   We have many tests for coercion, so I feel like the issue is more targeted than all coercion -- clearly we need to figure out what is going on and get some tests in place to make sure it doesn't happen again


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (Int --> String) [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2634643379

   I'm in the process of trying to track down another type coercion issue locally: 
   
   ```
   Internal error: Failed to match any signature, errors: Error during planning: The signature expected NativeType::String but received NativeType::Timestamp(Microsecond, Some("UTC"))
   ```
   The error message doesn't include the function name unfortunately (PR incoming soon).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (Int --> String) [datafusion]

Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on issue #14230:
URL: https://github.com/apache/datafusion/issues/14230#issuecomment-2634685064

   @Omega359 @alamb I've spent some time exploring the code and feel like I have a decent understanding of how it works. If it's helpful, I'd be happy to walk through it together. It shouldn't take long, and I think it could help clarify some of the points I've been trying to make. I tried to explain everything in detail in the PR, but I realize I might not have done the best job, so I'm happy to discuss further or clarify anything that's unclear.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org