You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/11/15 19:49:57 UTC
[I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
alamb opened a new issue, #13444:
URL: https://github.com/apache/datafusion/issues/13444
### Describe the bug
A regression appears to have been introduced in
- https://github.com/apache/datafusion/pull/13407
T
### To Reproduce
This used to work in DataFusion 42.0.0 but doesn't in DataFusion 43.0.0:
```
DataFusion CLI v43.0.0
> create table t as values (arrow_cast('Foo', 'Dictionary(Int32, LargeUtf8)'));
0 row(s) fetched.
Elapsed 0.013 seconds.
> select md5(column1) from t;
Arrow error: Compute error: Internal Error: Cannot cast Utf8View to StringArray of expected type
```
### Expected behavior
Query should complete
### Additional context
I found this while writing tests in https://github.com/apache/datafusion/pull/13443 -- Feel free to take those tests to work on this issue
Those tests fail like this:
```
External error: query failed: DataFusion error: Arrow error: Invalid argument error: column types must match schema types, expected LargeUtf8 but found Utf8 at column index 0
[SQL] select md5(ascii_1) from test_basic_operator;
at test_files/string/./string_query.slt.part:1381
at test_files/string/large_string.slt:93
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13444: Error running crypto functions on `Dictionary` arrays such as `md5`
URL: https://github.com/apache/datafusion/issues/13444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13444:
URL: https://github.com/apache/datafusion/issues/13444#issuecomment-2480120237
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13444:
URL: https://github.com/apache/datafusion/issues/13444#issuecomment-2480247473
I think adding casting from binary to string in TypeSignature::String and apply it to md5 is able to fix this too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13444:
URL: https://github.com/apache/datafusion/issues/13444#issuecomment-2480119872
I think this may have been a pre-existing issue. The core issue I think is that apparently md5 is required to return utf8: https://github.com/apache/datafusion/blob/6d8313ebc865f9bff007bfc04652f58b016cbc1b/datafusion/functions/src/crypto/basic.rs#L190 but the return_type function says it can return other types:
```Rust
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
use DataType::*;
Ok(match &arg_types[0] {
LargeUtf8 | LargeBinary => LargeUtf8,
Utf8View | Utf8 | Binary => Utf8,
Null => Null,
Dictionary(_, t) => match **t {
LargeUtf8 | LargeBinary => LargeUtf8,
Utf8 | Binary => Utf8,
Null => Null,
_ => {
return plan_err!(
"the md5 can only accept strings but got {:?}",
**t
);
}
},
other => {
return plan_err!(
"The md5 function can only accept strings. Got {other}"
);
}
})
}
```
Should be an easy fix to the return_type fn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Error running crypto functions on `Dictionary` arrays such as `md5` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13444:
URL: https://github.com/apache/datafusion/issues/13444#issuecomment-2486943799
I think I pushed a fix for this to @lamb 's repo as a PR there. I'll try and find it and submit the PR here instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org