You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/26 20:18:36 UTC
[GitHub] [arrow-nanoarrow] paleolimbot opened a new issue, #8: Implement `struct ArrowArrayBuilder`
paleolimbot opened a new issue, #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8
This will require a bit of code but is also the thing that most people need help with when generating C arrays (looping over some row-based structure and generating a column-based structure). One way to do this is a bunch of callables that are something like the reverse of SQLite's result interface:
```c
struct ArrowArrayBuilder {
ArrowErrorCode (*append_null)(struct ArrowArraybuilder*);
ArrowErrorCode (*append_int64)(struct ArrowArrayBuilder*, int64_t value);
ArrowErrorCode (*append_double)(struct ArrowArrayBuilder*, double value);
ArrowErrorCode (*append_string)(struct ArrowArrayBuilder*, struct ArrowStringView* value);
ArrowErrorCode (*append_blob)(struct ArrowArrayBuilder*, struct ArrowBufferView* value);
ArrowErrorCode (*append_array)(struct ArrowBufferView*, struct ArrowArray*, struct ArrowSchema*);
ArrowErrorCode (*borrow_schema)(struct ArrowBufferView*, struct ArrowSchema** schema_out);
ArrowErrorCode (*build_array)(struct ArrowBufferView*, struct ArrowArray* array_out, struct ArrowSchema* schema_out);
void (*release)(struct ArrowArrayBuilder*);
void* private_data;
};
```
Unike SQLite, I think this should error instead of coerce (i.e., `append_string()` on an integer builder errors and doesn't try to parse the string).
We could/should provide implementations for some array types, which I think only has to be one function. This
```c
ArrowErrorCode ArrowBuilderInit(struct ArrowBufferView* builder, struct ArrowSchema* schema);
```
Errors could be added as a `struct ArrowError*` arg to each method or as a `struct ArrowError` member to the `ArrowArrayBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195980100
Maybe the first step is not a builder class then...as per #5, a `struct ArrowArray` is already just a "bag of `struct ArrowBuffer`s" that can can be reserved/appended. Then we can do `ArrowInt32BuilderAppendInteger(struct ArrowArray* array, int32_t* values, int64_t n)` as a macro/inlined function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] lidavidm commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195974480
They could perhaps be Arrow type-oblivious macros that expand into a write into a particular buffer (and possibly a bounds-check/reserve call)? Then the builder is just a bag of 3 buffers, a length, and a null count, and there's no need for per-type variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #8: Implement element-wise appenders for `struct ArrowArray`s that we allocated
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1206564623
That's an excellent point, and David's "bag of buffers" comment makes a lot of sense. Type-specific appenders are definitely the way to go and I think after #5 we'll have what it takes to make the "accumulate a record batch from a schema defined at runtime" workflow a thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] lidavidm commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195959994
Looks reasonable to me. We may eventually want more conveniences, e.g. being able to append a range of values from a `int64_t*`, but this would work to start with. Also a `reserve` function may be useful (C++ has both `reserve` and `reserve_data` to deal with string/binary columns).
What would the API look like for lists, structs, and unions? I suppose you could emulate a list builder by pretending you're building an int32 array, a struct builder really is only a bitmap builder, and a union builder is a int8 builder where you never append nulls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] wesm commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
wesm commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195963131
I'm thinking that performance sensitive applications (especially those that might be migrating from C++ where anything that's in the Arrow builder headers is inlined) many of the builder APIs will need to be either macros or inline-able functions for the best performance as opposed to function pointers, at least when we are talking at the single-value level. @pitrou would probably have some advice about this (many performance critical APIs in CPython are implemented as C macros)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] pitrou commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1203706719
> One way to do this is a bunch of callables that are something like the reverse of SQLite's result interface
So this is for building arrays whose type is not strictly known at compile-time? Going through function pointers will make this slower than if you expose type-specific inline functions/macros, at least when appending one items at a time.
We have to think about the use cases for nanoarrow. Mostly, it will be used for bridging/converting with non-Arrow systems, and in this scheme people will probably have runtime switch/case statements to dispatch based on datatype anyway. So I'm not sure an untyped convention is really useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195967377
That's a good point...we need a system for that for the buffer builders too since those functions should be inlinable too (but it's inconvenient for readability if they're all sitting in nanoarrow.h).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] wesm commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
wesm commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195977036
I guess for the performance sensitive use case, I would focus on the reserve-then-unsafeappend usage pattern which we have in plenty of places in the Arrow codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot closed issue #8: Implement element-wise appenders for `struct ArrowArray`s that we allocated
Posted by GitBox <gi...@apache.org>.
paleolimbot closed issue #8: Implement element-wise appenders for `struct ArrowArray`s that we allocated
URL: https://github.com/apache/arrow-nanoarrow/issues/8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195962996
I'm not sure about the nested types...perhaps for the first PR I'll get the non-nested types in place and after that maybe it will be clearer whether the `struct ArrowArrayBuilder` needs to change to accommodate them or whether they're each their own thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #8: Implement `struct ArrowArrayBuilder`
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #8:
URL: https://github.com/apache/arrow-nanoarrow/issues/8#issuecomment-1195970110
I think with the callable approach those functions will all exist (i.e., `ArrowInt32BuilderAppendInteger()`)...maybe they live in something like src/nanoarrow/array_builder_inline.h.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org