You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2024/10/10 11:15:09 UTC
[PR] Macro for creating record batch from literal slice [datafusion]
timsaucer opened a new pull request, #12846:
URL: https://github.com/apache/datafusion/pull/12846
## Which issue does this PR close?
Closes #12574
## Rationale for this change
This macro allows for easy creation of a record batch, useful for prototype and unit testing. A user requested this feature.
## What changes are included in this PR?
Adds a macro to create a record batch. For example, the following creates a record batch with three arrays, a non nullable Int32, a nullable Float64, and a non nullable String.
```
let batch = create_batch!(
("a", Int32, vec![1, 2, 3]),
("b", Float64, vec![Some(4.0), None, Some(5.0)]),
("c", Utf8, vec!["alpha", "beta", "gamma"])
)?;
```
## Are these changes tested?
Tested both in the provided unit test and locally.
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#issuecomment-2408938367
Thanks agian -- this looks great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #12846:
URL: https://github.com/apache/datafusion/pull/12846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#discussion_r1797713584
##########
datafusion/common/src/test_util.rs:
##########
@@ -279,6 +279,83 @@ pub fn get_data_dir(
}
}
+#[macro_export]
+macro_rules! create_array {
+ (Boolean, $values: expr) => {
+ std::sync::Arc::new(arrow::array::BooleanArray::from($values))
+ };
+ (Int8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int8Array::from($values))
+ };
+ (Int16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int16Array::from($values))
+ };
+ (Int32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int32Array::from($values))
+ };
+ (Int64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int64Array::from($values))
+ };
+ (UInt8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt8Array::from($values))
+ };
+ (UInt16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt16Array::from($values))
+ };
+ (UInt32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt32Array::from($values))
+ };
+ (UInt64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt64Array::from($values))
+ };
+ (Float16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float16Array::from($values))
+ };
+ (Float32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float32Array::from($values))
+ };
+ (Float64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float64Array::from($values))
+ };
+ (Utf8, $values: expr) => {
Review Comment:
Added issue to track:
https://github.com/apache/datafusion/issues/12895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#discussion_r1796075363
##########
datafusion/common/src/test_util.rs:
##########
@@ -279,6 +279,83 @@ pub fn get_data_dir(
}
}
+#[macro_export]
+macro_rules! create_array {
+ (Boolean, $values: expr) => {
+ std::sync::Arc::new(arrow::array::BooleanArray::from($values))
+ };
+ (Int8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int8Array::from($values))
+ };
+ (Int16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int16Array::from($values))
+ };
+ (Int32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int32Array::from($values))
+ };
+ (Int64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int64Array::from($values))
+ };
+ (UInt8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt8Array::from($values))
+ };
+ (UInt16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt16Array::from($values))
+ };
+ (UInt32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt32Array::from($values))
+ };
+ (UInt64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt64Array::from($values))
+ };
+ (Float16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float16Array::from($values))
+ };
+ (Float32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float32Array::from($values))
+ };
+ (Float64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float64Array::from($values))
+ };
+ (Utf8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::StringArray::from($values))
+ };
+}
+
+/// Creates a record batch from literal slice of values, suitable for rapid
Review Comment:
this is beautiful. We can probably use it to clean up a bunch of the testing setup as well
##########
datafusion/common/src/test_util.rs:
##########
@@ -279,6 +279,83 @@ pub fn get_data_dir(
}
}
+#[macro_export]
+macro_rules! create_array {
+ (Boolean, $values: expr) => {
+ std::sync::Arc::new(arrow::array::BooleanArray::from($values))
+ };
+ (Int8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int8Array::from($values))
+ };
+ (Int16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int16Array::from($values))
+ };
+ (Int32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int32Array::from($values))
+ };
+ (Int64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int64Array::from($values))
+ };
+ (UInt8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt8Array::from($values))
+ };
+ (UInt16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt16Array::from($values))
+ };
+ (UInt32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt32Array::from($values))
+ };
+ (UInt64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt64Array::from($values))
+ };
+ (Float16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float16Array::from($values))
+ };
+ (Float32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float32Array::from($values))
+ };
+ (Float64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float64Array::from($values))
+ };
+ (Utf8, $values: expr) => {
Review Comment:
There are a bunch of other types that might be worth supporting too `LargeUtf8`, `Utf8View`, etc
Full list here:
https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html
We could definitely do it as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#discussion_r1797443813
##########
datafusion/common/src/test_util.rs:
##########
@@ -333,4 +410,15 @@ mod tests {
let res = parquet_test_data();
assert!(PathBuf::from(res).is_dir());
}
+
+ #[test]
+ fn test_create_record_batch() {
+ let batch = create_batch!(
+ ("a", Int32, vec![1, 2, 3]),
+ ("b", Float64, vec![Some(4.0), None, Some(5.0)]),
+ ("c", Utf8, vec!["alpha", "beta", "gamma"])
+ );
Review Comment:
nice!
##########
datafusion/common/src/test_util.rs:
##########
@@ -333,4 +410,15 @@ mod tests {
let res = parquet_test_data();
assert!(PathBuf::from(res).is_dir());
}
+
+ #[test]
+ fn test_create_record_batch() {
+ let batch = create_batch!(
+ ("a", Int32, vec![1, 2, 3]),
+ ("b", Float64, vec![Some(4.0), None, Some(5.0)]),
+ ("c", Utf8, vec!["alpha", "beta", "gamma"])
+ );
+
+ assert!(batch.is_ok());
Review Comment:
should we assert the contents too?
##########
datafusion/common/src/test_util.rs:
##########
@@ -279,6 +279,83 @@ pub fn get_data_dir(
}
}
+#[macro_export]
+macro_rules! create_array {
+ (Boolean, $values: expr) => {
+ std::sync::Arc::new(arrow::array::BooleanArray::from($values))
+ };
+ (Int8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int8Array::from($values))
+ };
+ (Int16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int16Array::from($values))
+ };
+ (Int32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int32Array::from($values))
+ };
+ (Int64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int64Array::from($values))
+ };
+ (UInt8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt8Array::from($values))
+ };
+ (UInt16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt16Array::from($values))
+ };
+ (UInt32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt32Array::from($values))
+ };
+ (UInt64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt64Array::from($values))
+ };
+ (Float16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float16Array::from($values))
+ };
+ (Float32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float32Array::from($values))
+ };
+ (Float64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float64Array::from($values))
+ };
+ (Utf8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::StringArray::from($values))
+ };
+}
+
+/// Creates a record batch from literal slice of values, suitable for rapid
+/// testing and development.
+///
+/// Example:
+/// ```
+/// use datafusion_common::{create_batch, create_array};
+/// let batch = create_batch!(
+/// ("a", Int32, vec![1, 2, 3]),
+/// ("b", Float64, vec![Some(4.0), None, Some(5.0)]),
+/// ("c", Utf8, vec!["alpha", "beta", "gamma"])
+/// );
+/// ```
+#[macro_export]
Review Comment:
does this affect how the macro is imported? do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#discussion_r1797713188
##########
datafusion/common/src/test_util.rs:
##########
@@ -279,6 +279,83 @@ pub fn get_data_dir(
}
}
+#[macro_export]
+macro_rules! create_array {
+ (Boolean, $values: expr) => {
+ std::sync::Arc::new(arrow::array::BooleanArray::from($values))
+ };
+ (Int8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int8Array::from($values))
+ };
+ (Int16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int16Array::from($values))
+ };
+ (Int32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int32Array::from($values))
+ };
+ (Int64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Int64Array::from($values))
+ };
+ (UInt8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt8Array::from($values))
+ };
+ (UInt16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt16Array::from($values))
+ };
+ (UInt32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt32Array::from($values))
+ };
+ (UInt64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::UInt64Array::from($values))
+ };
+ (Float16, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float16Array::from($values))
+ };
+ (Float32, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float32Array::from($values))
+ };
+ (Float64, $values: expr) => {
+ std::sync::Arc::new(arrow::array::Float64Array::from($values))
+ };
+ (Utf8, $values: expr) => {
+ std::sync::Arc::new(arrow::array::StringArray::from($values))
+ };
+}
+
+/// Creates a record batch from literal slice of values, suitable for rapid
+/// testing and development.
+///
+/// Example:
+/// ```
+/// use datafusion_common::{create_batch, create_array};
+/// let batch = create_batch!(
+/// ("a", Int32, vec![1, 2, 3]),
+/// ("b", Float64, vec![Some(4.0), None, Some(5.0)]),
+/// ("c", Utf8, vec!["alpha", "beta", "gamma"])
+/// );
+/// ```
+#[macro_export]
Review Comment:
I think we do need it so we can use it in downstream repos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#issuecomment-2408622238
I put in a couple of small additions per feedback here and on the arrow-rs issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#discussion_r1797729997
##########
datafusion/common/src/test_util.rs:
##########
@@ -333,4 +410,15 @@ mod tests {
let res = parquet_test_data();
assert!(PathBuf::from(res).is_dir());
}
+
+ #[test]
+ fn test_create_record_batch() {
+ let batch = create_batch!(
+ ("a", Int32, vec![1, 2, 3]),
+ ("b", Float64, vec![Some(4.0), None, Some(5.0)]),
+ ("c", Utf8, vec!["alpha", "beta", "gamma"])
+ );
+
+ assert!(batch.is_ok());
Review Comment:
Added in last push
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Macro for creating record batch from literal slice [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12846:
URL: https://github.com/apache/datafusion/pull/12846#issuecomment-2404978801
Maybe this is worth moving upstream to arrow-rs as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org