You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2024/10/12 14:43:29 UTC
[I] Import downstream macro for easy record batch creation [arrow-rs]
timsaucer opened a new issue, #6553:
URL: https://github.com/apache/arrow-rs/issues/6553
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In https://github.com/apache/datafusion/pull/12846 we added a macro to easily create a record batch for use in building up unit tests. It would be nice to move this upstream to this repo.
**Describe the solution you'd like**
Port code and optionally add remaining data types.
**Describe alternatives you've considered**
None
**Additional context**
https://github.com/apache/datafusion/pull/12846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "ByteBaker (via GitHub)" <gi...@apache.org>.
ByteBaker commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2423257712
I'm a bit confused about the definition of this macro (and subsequent import implications).
1. If I wanna use it inside `arrow-array::record_batch` (for tests), I need to use `crate::<some type>` in the macro definition.
2. Doing (1) would mean I cannot use it in code outside `arrow-array` because for that, I must have `arrow_array::<some type>` in the macro definition.
3. **1** & **2** are mutually exclusive.
For example:
```rust
/// Works where `arrow_array` is a dependency
#[macro_export]
macro_rules! create_array {
(@array_of Boolean) => { arrow_array::BooleanArray };
// Other types here
($ty: tt, [$($values: expr),*]) => {
std::sync::Arc::new(<crate::create_array!(@array_of $ty)>::from(vec![$($values),*]))
};
}
```
This works inside `arrow_array`
```rust
#[macro_export]
macro_rules! create_array {
(@array_of Boolean) => { crate::BooleanArray };
// Other types here
($ty: tt, [$($values: expr),*]) => {
std::sync::Arc::new(<crate::create_array!(@array_of $ty)>::from(vec![$($values),*]))
};
}
```
@alamb @timsaucer some help please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6553: Add `record_batch!` macro for easy record batch creation
URL: https://github.com/apache/arrow-rs/issues/6553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2414373607
> Or do we want something that allows us to write a template-like, visually intuitive structure for creating an RB? For instance:
Thanks @ByteBaker -- maybe you can check out the implementation that @timsaucer put into DataFusion in https://github.com/apache/datafusion/pull/12846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "ByteBaker (via GitHub)" <gi...@apache.org>.
ByteBaker commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2409051978
@alamb this is the basic idea I have. Lemme know if I should change this.
```rust
let schema: Arc<Schema> = ...;
let options: RecordBatchOptions = ...;
let array: Arc<Int32Array> = ...; // Can have more arrays
record_batch!(schema); // Creates empty RecordBatch
record_batch!(schema, [array]); // Returns a result
record_batch!(schema, [array], options); // Creates w/ options, returns a result
```
Or do we want something that allows us to write a template-like, visually intuitive structure for creating an RB? For instance:
```rust
record_batch!{
// Always required
[
"id", Int32, false // More fields can go in next lines
],
// Following bit be optional.
// Not giving data would create an empty RecordBatch
[
Int32Array[1, 2, 3, 4, 5] // More arrays can go in next lines
],
// `RecordBatchOptions` with `match_field_names` = false & `row_count` = Some(10)
// This bit is optional, and the absence will just use the default options
{true, 10}
}
```
Also, how do we reconcile the return types in this case?
- Using just `Schema` would create a `RecordBatch` directly
- Any other combination would return `Result<RecordBatch>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import downstream macro for easy record batch creation [arrow-rs]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2408621933
I incorporated your recommendation in the last push of the PR in datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2408938769
I think this is a good first issue for someone who wants to try their hand out at contributing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Import downstream macro for easy record batch creation [arrow-rs]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2408589547
Hi, I like this idea. How about renaming this macro to `record_batch!()` like `vec!()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2408938555
> I incorporated your recommendation in the last push of the PR in datafusion.
I think it looks really nice -- thank you @timsaucer and @Xuanwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "ByteBaker (via GitHub)" <gi...@apache.org>.
ByteBaker commented on issue #6553:
URL: https://github.com/apache/arrow-rs/issues/6553#issuecomment-2409044873
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `record_batch!` macro for easy record batch creation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6553: Add `record_batch!` macro for easy record batch creation
URL: https://github.com/apache/arrow-rs/issues/6553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org