You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/04/10 14:50:21 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #5946: Remove transmute in datafusion-proto
alamb opened a new pull request, #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946
# Which issue does this PR close?
Related to https://github.com/apache/arrow-datafusion/issues/5717
# Rationale for this change
During the review of https://github.com/apache/arrow-datafusion/pull/5775 I noticed there was unsafe code in `datafusion-proto` for seemingly unecessary reasons. See https://github.com/apache/arrow-datafusion/pull/5775#discussion_r1161492804
# What changes are included in this PR?
1. Remove `unsafe` code.
2. Add boundary tests to ensure substrait roundtrip casting works
# Are these changes tested?
Yes new tests are added
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946#discussion_r1161789045
##########
datafusion/substrait/src/logical_plan/consumer.rs:
##########
@@ -453,7 +453,12 @@ pub async fn from_substrait_sorts(
let asc_nullfirst = match &s.sort_kind {
Some(k) => match k {
Direction(d) => {
- let direction: SortDirection = unsafe { ::std::mem::transmute(*d) };
+ let Some(direction) = SortDirection::from_i32(*d) else {
Review Comment:
The point of the PR is to remove all uses of `transmute` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waynexia commented on pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946#issuecomment-1501970837
Never mind 😃 Fix in #5947, PTAL 😉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waynexia commented on pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946#issuecomment-1501962028
Oops, I'm late 🤣 I'll file a fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener merged pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener merged PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waynexia commented on a diff in pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on code in PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946#discussion_r1161823565
##########
datafusion/substrait/src/logical_plan/producer.rs:
##########
@@ -1285,3 +1283,62 @@ fn substrait_field_ref(index: usize) -> Result<Expression> {
}))),
})
}
+
+#[cfg(test)]
+mod test {
+ use crate::logical_plan::consumer::from_substrait_literal;
+
+ use super::*;
+
+ #[test]
+ fn round_trip_literals() -> Result<()> {
+ round_trip_literal(ScalarValue::Boolean(None))?;
Review Comment:
This should fail, I find I forget to match Boolean Null
##########
datafusion/substrait/src/logical_plan/producer.rs:
##########
@@ -1285,3 +1283,62 @@ fn substrait_field_ref(index: usize) -> Result<Expression> {
}))),
})
}
+
+#[cfg(test)]
+mod test {
+ use crate::logical_plan::consumer::from_substrait_literal;
+
+ use super::*;
+
+ #[test]
+ fn round_trip_literals() -> Result<()> {
+ round_trip_literal(ScalarValue::Boolean(None))?;
+ round_trip_literal(ScalarValue::Boolean(Some(true)))?;
+ round_trip_literal(ScalarValue::Boolean(Some(false)))?;
+
+ round_trip_literal(ScalarValue::Int8(None))?;
+ round_trip_literal(ScalarValue::Int8(Some(i8::MIN)))?;
+ round_trip_literal(ScalarValue::Int8(Some(i8::MAX)))?;
+ round_trip_literal(ScalarValue::UInt8(None))?;
+ round_trip_literal(ScalarValue::UInt8(Some(u8::MIN)))?;
+ round_trip_literal(ScalarValue::UInt8(Some(u8::MAX)))?;
+
+ round_trip_literal(ScalarValue::Int16(None))?;
+ round_trip_literal(ScalarValue::Int16(Some(i16::MIN)))?;
+ round_trip_literal(ScalarValue::Int16(Some(i16::MAX)))?;
+ round_trip_literal(ScalarValue::UInt16(None))?;
+ round_trip_literal(ScalarValue::UInt16(Some(u16::MIN)))?;
+ round_trip_literal(ScalarValue::UInt16(Some(u16::MAX)))?;
+
+ round_trip_literal(ScalarValue::Int32(None))?;
+ round_trip_literal(ScalarValue::Int32(Some(i32::MIN)))?;
+ round_trip_literal(ScalarValue::Int32(Some(i32::MAX)))?;
+ round_trip_literal(ScalarValue::UInt32(None))?;
+ round_trip_literal(ScalarValue::UInt32(Some(u32::MIN)))?;
+ round_trip_literal(ScalarValue::UInt32(Some(u32::MAX)))?;
+
+ round_trip_literal(ScalarValue::Int64(None))?;
+ round_trip_literal(ScalarValue::Int64(Some(i64::MIN)))?;
+ round_trip_literal(ScalarValue::Int64(Some(i64::MAX)))?;
+ round_trip_literal(ScalarValue::UInt64(None))?;
+ round_trip_literal(ScalarValue::UInt64(Some(u64::MIN)))?;
+ round_trip_literal(ScalarValue::UInt64(Some(u64::MAX)))?;
+
+ Ok(())
+ }
+
+ fn round_trip_literal(scalar: ScalarValue) -> Result<()> {
+ println!("Checking round trip of {:?}", scalar);
+
+ let scalar = ScalarValue::Int32(Some(i32::MAX));
Review Comment:
It shadows the input
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on pull request #5946: Remove `unsafe` code (`transmute`) in datafusion-proto
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #5946:
URL: https://github.com/apache/arrow-datafusion/pull/5946#issuecomment-1501966991
> Oops, I'm late 🤣 I'll file a fix
My fault🥲, I merged it due to feel there is no problem.
I need to pay attention in the future, try to wait for enough time for each PR.
This is a good lesson for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org