You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "codetyri0n (via GitHub)" <gi...@apache.org> on 2025/12/09 12:44:00 UTC

[PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

codetyri0n opened a new pull request, #19235:
URL: https://github.com/apache/datafusion/pull/19235

   ## Which issue does this PR close?
   part of #12725 
   ## Rationale for this change
   moving `hex` away from a user defined signature as it is out of scope and added a UT for checking correct number of args
   ## Are these changes tested?
   existing tests 
   ## Are there any user-facing changes?
   yes
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610506181


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   Actually this signature can just be:
   
   ```rust
   Coercion::new_exact(TypeSignatureClass::Binary);
   ```
   
   It seems the implementation can already handle FixedSizeBinary's so theres no need to cast those to Binary/LargeBinary type 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610510943


##########
datafusion/sqllogictest/test_files/spark/math/hex.slt:
##########
@@ -48,3 +48,26 @@ SELECT hex(column1) FROM t_utf8view;
 666F6F
 NULL
 666F6F62617262617A
+
+statement ok
+CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
+
+query T
+SELECT hex(column1) FROM t_largebinary;
+----
+68656C6C6F
+NULL
+776F726C64
+
+query T
+SELECT hex(arrow_cast(arrow_cast('!hallucinations!', 'Binary'), 'LargeBinary'));
+----
+2168616C6C7563696E6174696F6E7321

Review Comment:
   Can remove this as seems redundant with the LargeBinary test at the bottom 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2609584675


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   > CREATE TABLE t_largebinary AS VALUES
     (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
     (NULL::LargeBinary),
     (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
     (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
     (NULL::LargeBinary),
     (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
     (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
     (NULL::LargeBinary),
     (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));CREATE TABLE t_largebinary AS VALUES
     (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')),
     (NULL::LargeBinary),
     (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
   This feature is not implemented: Unsupported SQL type LargeBinary
   This feature is not implemented: Unsupported SQL type LargeBinary
   This feature is not implemented: Unsupported SQL type LargeBinary
   This feature is not implemented: Unsupported SQL type LargeBinary
   
   Getting the above in datafusion cli



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610499987


##########
datafusion/expr-common/src/signature.rs:
##########
@@ -337,7 +337,7 @@ pub enum TypeSignatureClass {
     Float,
     Decimal,
     Numeric,
-    /// Encompasses both the native Binary as well as arbitrarily sized FixedSizeBinary types
+    /// Encompasses Binary, FixedSizeBinary, and LargeBinary types

Review Comment:
   ```suggestion
       /// Encompasses both the native Binary/LargeBinary types as well as arbitrarily sized FixedSizeBinary types
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2602734976


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],
+            NativeType::String,

Review Comment:
   ```suggestion
               NativeType::Binary,
   ```
   ?!



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],

Review Comment:
   ```suggestion
               vec![TypeSignatureClass::Numeric],
   ```
   Numeric includes Integer, so no need to list it explicitly.



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   This does not support `LargeBinary` type.
   The old version supported it - https://github.com/apache/datafusion/pull/19235/files#diff-6d740fb90f0b078d335d596dcb8211ae4a32a6df40fcc7aa60ee779b3f800bc6L106



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2605282326


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -421,4 +411,21 @@ mod test {
 
         assert_eq!(string_array, &expected_array);
     }
+
+    #[test]
+    fn test_hex_multiple_arguments() {

Review Comment:
   Could we move this to SLT instead?



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +55,35 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );

Review Comment:
   ```suggestion
           let string = Coercion::new_exact(
               TypeSignatureClass::Native(logical_string()),
           );
   ```
   
   No need to coerce from itself



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   This seems a bug if `logical_binary()` doesn't cover LargeBinary; can we fix this at the root cause instead of having a workaround?



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -240,6 +220,16 @@ pub fn compute_hex(
 
                 Ok(ColumnarValue::Array(Arc::new(hexed)))
             }
+            DataType::LargeBinary => {

Review Comment:
   Can we add an SLT test for largebinary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610169298


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   my idea was to use explain to determine if LargeBinary comes under native binary using a query in which I cast like - `SELECT hex(arrow_cast(arrow_cast('test', 'Binary'), 'LargeBinary'));` which did not give anything conclusive.
   so I tested out an slt with `SELECT hex(arrow_cast('test', 'LargeBinary')) as lar_b;` after commenting out
    ` TypeSignature::Exact(vec![DataType::LargeBinary]),` ---- which passed without any failure.
    
   to conclude - we can now be sure that LargeBinary is encompassed under the native Binary right? 
   lmk your thoughts @martin-g @Jefffrey 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610515069


##########
datafusion/sqllogictest/test_files/spark/math/hex.slt:
##########
@@ -48,3 +48,26 @@ SELECT hex(column1) FROM t_utf8view;
 666F6F
 NULL
 666F6F62617262617A
+
+statement ok
+CREATE TABLE t_largebinary as VALUES (arrow_cast(arrow_cast('hello', 'Binary'), 'LargeBinary')), (NULL), (arrow_cast(arrow_cast('world', 'Binary'), 'LargeBinary'));
+
+query T
+SELECT hex(column1) FROM t_largebinary;
+----
+68656C6C6F
+NULL
+776F726C64

Review Comment:
   ```suggestion
   query T
   SELECT hex(column1) FROM VALUES (arrow_cast('hello', 'LargeBinary')), (NULL), (arrow_cast('world', 'LargeBinary'));
   ----
   68656C6C6F
   NULL
   776F726C64
   ```
   
   So we don't need to worry about cleaning up the table at the end. Also avoid need for cast from string -> binary -> largebinary, just convert directly to largebinary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2606548964


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   My comment was about `vec![TypeSignatureClass::Binary],`, not `logical_binary()`.
   https://github.com/apache/datafusion/blob/2a08013af3ccf703bee202c959b40bb0d35bdea1/datafusion/expr-common/src/signature.rs#L340-L341
   The comment here mentions (the native) Binary and FixedSizeBinary but nothing about LargeBinary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2602762177


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],
+            NativeType::String,

Review Comment:
   > ?!
   
   ahh 🤦🏽‍♂️  (thank you : ) )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #19235:
URL: https://github.com/apache/datafusion/pull/19235


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2606468943


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   im trying to look at where that may be but cant seem to pinpoint beyond https://github.com/apache/datafusion/blob/main/datafusion/common/src/types/builtin.rs#L77



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610305985


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   OK! Then the comment at https://github.com/apache/datafusion/blob/2a08013af3ccf703bee202c959b40bb0d35bdea1/datafusion/expr-common/src/signature.rs#L340-L341 could be updated to mention LargeBinary too!
   Please add the SLT test to the suite!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2610169298


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   my idea was to use explain analyze to determine if LargeBinary comes under native binary using a query in which I cast like - `SELECT hex(arrow_cast(arrow_cast('test', 'Binary'), 'LargeBinary'));` which did not give anything conclusive.
   so I tested out an slt with `SELECT hex(arrow_cast('test', 'LargeBinary')) as lar_b;` after commenting out
    ` TypeSignature::Exact(vec![DataType::LargeBinary]),` ---- which passed without any failure.
    
   to conclude - we can now be sure that LargeBinary is encompassed under the native Binary right? 
   lmk your thoughts @martin-g @Jefffrey 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "codetyri0n (via GitHub)" <gi...@apache.org>.
codetyri0n commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2607041934


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   ill do some testing locally and see what im getting



##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   ill do some testing locally and see what im getting then



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2606854236


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   I'm fairly sure it encompasses LargeBinary too, but if we had a test to confirm this it would be nice



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#discussion_r2609789430


##########
datafusion/spark/src/function/math/hex.rs:
##########
@@ -52,8 +54,33 @@ impl Default for SparkHex {
 
 impl SparkHex {
     pub fn new() -> Self {
+        let int64 = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_int64()),
+            vec![TypeSignatureClass::Integer, TypeSignatureClass::Numeric],
+            NativeType::Int64,
+        );
+        let string = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_string()),
+            vec![TypeSignatureClass::Native(logical_string())],
+            NativeType::String,
+        );
+        let binary = Coercion::new_implicit(
+            TypeSignatureClass::Native(logical_binary()),
+            vec![TypeSignatureClass::Binary],

Review Comment:
   ```
   > CREATE TABLE binary_data (id INTEGER, data BYTEA);
   0 row(s) fetched.
   Elapsed 0.002 seconds.
   
   > INSERT INTO binary_data (id, data) VALUES (1, X'48656C6C6F');
   +-------+
   | count |
   +-------+
   | 1     |
   +-------+
   1 row(s) fetched.
   Elapsed 0.017 seconds.
   
   > select * from binary_data;
   +----+------------+
   | id | data       |
   +----+------------+
   | 1  | 48656c6c6f |
   +----+------------+
   1 row(s) fetched.
   Elapsed 0.003 seconds.
   
   > select arrow_cast(data, 'LargeBinary') from binary_data;
   +--------------------------------------------------+
   | arrow_cast(binary_data.data,Utf8("LargeBinary")) |
   +--------------------------------------------------+
   | 48656c6c6f                                       |
   +--------------------------------------------------+
   1 row(s) fetched.
   Elapsed 0.001 seconds.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] [datafusion-spark]: Refactor hex's signature away from user_defined [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19235:
URL: https://github.com/apache/datafusion/pull/19235#issuecomment-3644990891

   Thanks @codetyri0n & @martin-g 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org