You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "scovich (via GitHub)" <gi...@apache.org> on 2025/08/20 17:14:38 UTC
[PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
scovich opened a new pull request, #8185:
URL: https://github.com/apache/arrow-rs/pull/8185
# Which issue does this PR close?
- Closes https://github.com/apache/arrow-rs/issues/8182 (directly)
- Closes https://github.com/apache/arrow-rs/issues/8170 (as a side effect)
- Closes https://github.com/apache/arrow-rs/issues/8180 (as a side effect)
# Rationale for this change
Make finish/rollback handling simpler, robust and uniform by pulling it inside `ParentState` (instead of each builder doing it manually) and making it fully eager (instead of a mix of eager and lazy).
# What changes are included in this PR?
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
# Are these changes tested?
Yes, existing unit tests cover it (including some that needed adjustment due to behavior changes), along with some new testing.
# Are there any user-facing changes?
`ObjectBuilder` methods `new_list` and `new_object` now panic if a duplicate field name is provided, and new fallible versions `try_new_list` and `try_new_object` are provided.
`ObjectBuilder::finish` signature remains fallible, but it always returns `Result::Ok`
* TODO: https://github.com/apache/arrow-rs/issues/8184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3207591252
FYI @codephage2020 and @klion26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3207813918
@alamb -- this should be ready to merge unless we need more reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3207720194
> As I understand it, this prepares the signatures for a type of `VariantBuilder` that has immutable metadata (so that trying to add a field without an existing dictionary entry will fail) but not yet done
Correct! Duplicate field name (normal builder) or unknown field name (read-only builder) will both fail in the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3207833751
> @alamb -- this should be ready to merge unless we need more reviews
I think it would be nice to let @codephage2020 / @klion26 a chance to review too as I think it is outside their working hours now. I'll plan to merge tomorrow morning eastern time unless anyone else wants a chance to review
We can, of course, also address any comments as follow on PRs too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3211186453
Thank you @scovich and @klion26 and @codephage2020 for the reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#issuecomment-3207514671
@alamb -- this one should be ready for review. It's mostly independent of
* https://github.com/apache/arrow-rs/pull/8187/
... but there will be some merge conflicts due to the name changes. I don't think it matters which one merges first, the rebase will be straightforward either way.
Once both of these are in, I can put up the next PR pretty quickly, that makes `ValueBuilder` work directly with `ParentState`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#discussion_r2288977440
##########
parquet-variant/src/builder.rs:
##########
@@ -589,72 +582,190 @@ impl<S: AsRef<str>> Extend<S> for MetadataBuilder {
/// parent, and we cannot "split" a mutable reference across two objects (parent state and the child
/// builder that uses it). So everything has to be here. Rust layout optimizations should treat the
/// variants as a union, so that accessing a `buffer` or `metadata_builder` is branch-free.
+#[derive(Debug)]
enum ParentState<'a> {
Variant {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
+ finished: bool,
},
List {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
- parent_value_offset_base: usize,
+ saved_metadata_builder_dict_size: usize,
offsets: &'a mut Vec<usize>,
+ saved_offsets_size: usize,
+ finished: bool,
},
Object {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
fields: &'a mut IndexMap<u32, usize>,
- field_name: &'a str,
- parent_value_offset_base: usize,
+ saved_fields_size: usize,
+ finished: bool,
},
}
-impl ParentState<'_> {
- fn buffer(&mut self) -> &mut ValueBuffer {
- match self {
- ParentState::Variant { buffer, .. } => buffer,
- ParentState::List { buffer, .. } => buffer,
- ParentState::Object { buffer, .. } => buffer,
+impl<'a> ParentState<'a> {
+ fn variant(buffer: &'a mut ValueBuffer, metadata_builder: &'a mut MetadataBuilder) -> Self {
+ ParentState::Variant {
+ saved_buffer_offset: buffer.offset(),
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ buffer,
+ metadata_builder,
+ finished: false,
+ }
+ }
+
+ fn list(
+ buffer: &'a mut ValueBuffer,
+ metadata_builder: &'a mut MetadataBuilder,
+ offsets: &'a mut Vec<usize>,
+ saved_parent_buffer_offset: usize,
+ ) -> Self {
+ // The saved_parent_buffer_offset is the buffer size as of when the parent builder was
+ // constructed. The saved_buffer_offset is the buffer size as of now (when a child builder
+ // is created). The variant field_offset entry for this list element is their difference.
+ let saved_buffer_offset = buffer.offset();
+ let saved_offsets_size = offsets.len();
+ offsets.push(saved_buffer_offset - saved_parent_buffer_offset);
+
+ ParentState::List {
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ saved_buffer_offset,
+ saved_offsets_size,
+ metadata_builder,
+ buffer,
+ offsets,
+ finished: false,
}
}
+ fn object(
Review Comment:
nit: could call this `try_object` to reflect it can return `Result`
##########
parquet-variant/src/builder.rs:
##########
@@ -589,72 +582,190 @@ impl<S: AsRef<str>> Extend<S> for MetadataBuilder {
/// parent, and we cannot "split" a mutable reference across two objects (parent state and the child
/// builder that uses it). So everything has to be here. Rust layout optimizations should treat the
/// variants as a union, so that accessing a `buffer` or `metadata_builder` is branch-free.
+#[derive(Debug)]
enum ParentState<'a> {
Variant {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
+ finished: bool,
},
List {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
- parent_value_offset_base: usize,
+ saved_metadata_builder_dict_size: usize,
offsets: &'a mut Vec<usize>,
+ saved_offsets_size: usize,
+ finished: bool,
},
Object {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
fields: &'a mut IndexMap<u32, usize>,
- field_name: &'a str,
- parent_value_offset_base: usize,
+ saved_fields_size: usize,
+ finished: bool,
},
}
-impl ParentState<'_> {
- fn buffer(&mut self) -> &mut ValueBuffer {
- match self {
- ParentState::Variant { buffer, .. } => buffer,
- ParentState::List { buffer, .. } => buffer,
- ParentState::Object { buffer, .. } => buffer,
+impl<'a> ParentState<'a> {
+ fn variant(buffer: &'a mut ValueBuffer, metadata_builder: &'a mut MetadataBuilder) -> Self {
+ ParentState::Variant {
+ saved_buffer_offset: buffer.offset(),
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ buffer,
+ metadata_builder,
+ finished: false,
+ }
+ }
+
+ fn list(
+ buffer: &'a mut ValueBuffer,
+ metadata_builder: &'a mut MetadataBuilder,
+ offsets: &'a mut Vec<usize>,
+ saved_parent_buffer_offset: usize,
+ ) -> Self {
+ // The saved_parent_buffer_offset is the buffer size as of when the parent builder was
+ // constructed. The saved_buffer_offset is the buffer size as of now (when a child builder
+ // is created). The variant field_offset entry for this list element is their difference.
+ let saved_buffer_offset = buffer.offset();
+ let saved_offsets_size = offsets.len();
+ offsets.push(saved_buffer_offset - saved_parent_buffer_offset);
+
+ ParentState::List {
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ saved_buffer_offset,
+ saved_offsets_size,
+ metadata_builder,
+ buffer,
+ offsets,
+ finished: false,
}
}
+ fn object(
+ buffer: &'a mut ValueBuffer,
+ metadata_builder: &'a mut MetadataBuilder,
+ fields: &'a mut IndexMap<u32, usize>,
+ saved_parent_buffer_offset: usize,
+ field_name: &str,
+ validate_unique_fields: bool,
+ ) -> Result<Self, ArrowError> {
+ // The saved_parent_buffer_offset is the buffer size as of when the parent builder was
+ // constructed. The saved_buffer_offset is the buffer size as of now (when a child builder
+ // is created). The variant field_offset entry for this field is their difference.
+ let saved_buffer_offset = buffer.offset();
+ let saved_fields_size = fields.len();
+ let saved_metadata_builder_dict_size = metadata_builder.num_field_names();
+ let field_id = metadata_builder.upsert_field_name(field_name);
+ let field_start = saved_buffer_offset - saved_parent_buffer_offset;
+ if fields.insert(field_id, field_start).is_some() && validate_unique_fields {
+ return Err(ArrowError::InvalidArgumentError(format!(
+ "Duplicate field name: {field_name}"
+ )));
+ }
+
+ Ok(ParentState::Object {
+ saved_metadata_builder_dict_size,
+ saved_buffer_offset,
+ saved_fields_size,
+ buffer,
+ metadata_builder,
+ fields,
+ finished: false,
+ })
+ }
+
+ fn buffer(&mut self) -> &mut ValueBuffer {
+ self.buffer_and_metadata_builder().0
+ }
+
fn metadata_builder(&mut self) -> &mut MetadataBuilder {
+ self.buffer_and_metadata_builder().1
+ }
+
+ fn saved_buffer_offset(&mut self) -> usize {
match self {
ParentState::Variant {
- metadata_builder, ..
- } => metadata_builder,
- ParentState::List {
- metadata_builder, ..
- } => metadata_builder,
- ParentState::Object {
- metadata_builder, ..
- } => metadata_builder,
+ saved_buffer_offset,
+ ..
+ }
+ | ParentState::List {
+ saved_buffer_offset,
+ ..
+ }
+ | ParentState::Object {
+ saved_buffer_offset,
+ ..
+ } => *saved_buffer_offset,
}
}
- // Performs any parent-specific aspects of finishing, after the child has appended all necessary
- // bytes to the parent's value buffer. ListBuilder records the new value's starting offset;
- // ObjectBuilder associates the new value's starting offset with its field id; VariantBuilder
- // doesn't need anything special.
- fn finish(&mut self, starting_offset: usize) {
+ fn is_finished(&mut self) -> &mut bool {
+ match self {
+ ParentState::Variant { finished, .. }
+ | ParentState::List { finished, .. }
+ | ParentState::Object { finished, .. } => finished,
+ }
+ }
+
+ // Mark the insertion as having succeeded.
+ fn finish(&mut self) {
+ *self.is_finished() = true
+ }
+
+ // Performs any parent-specific aspects of rolling back a builder if an insertion failed.
+ fn rollback(&mut self) {
+ if *self.is_finished() {
+ return;
+ }
+
+ // All builders need to revert the buffers
+ match self {
+ ParentState::Variant {
+ buffer,
+ saved_buffer_offset,
+ metadata_builder,
+ saved_metadata_builder_dict_size,
+ ..
+ }
+ | ParentState::List {
+ buffer,
+ saved_buffer_offset,
+ metadata_builder,
+ saved_metadata_builder_dict_size,
+ ..
+ }
+ | ParentState::Object {
+ buffer,
+ saved_buffer_offset,
+ metadata_builder,
+ saved_metadata_builder_dict_size,
+ ..
+ } => {
+ buffer.inner_mut().truncate(*saved_buffer_offset);
Review Comment:
this is very nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185#discussion_r2289016779
##########
parquet-variant/src/builder.rs:
##########
@@ -589,72 +582,190 @@ impl<S: AsRef<str>> Extend<S> for MetadataBuilder {
/// parent, and we cannot "split" a mutable reference across two objects (parent state and the child
/// builder that uses it). So everything has to be here. Rust layout optimizations should treat the
/// variants as a union, so that accessing a `buffer` or `metadata_builder` is branch-free.
+#[derive(Debug)]
enum ParentState<'a> {
Variant {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
+ finished: bool,
},
List {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
- parent_value_offset_base: usize,
+ saved_metadata_builder_dict_size: usize,
offsets: &'a mut Vec<usize>,
+ saved_offsets_size: usize,
+ finished: bool,
},
Object {
buffer: &'a mut ValueBuffer,
+ saved_buffer_offset: usize,
metadata_builder: &'a mut MetadataBuilder,
+ saved_metadata_builder_dict_size: usize,
fields: &'a mut IndexMap<u32, usize>,
- field_name: &'a str,
- parent_value_offset_base: usize,
+ saved_fields_size: usize,
+ finished: bool,
},
}
-impl ParentState<'_> {
- fn buffer(&mut self) -> &mut ValueBuffer {
- match self {
- ParentState::Variant { buffer, .. } => buffer,
- ParentState::List { buffer, .. } => buffer,
- ParentState::Object { buffer, .. } => buffer,
+impl<'a> ParentState<'a> {
+ fn variant(buffer: &'a mut ValueBuffer, metadata_builder: &'a mut MetadataBuilder) -> Self {
+ ParentState::Variant {
+ saved_buffer_offset: buffer.offset(),
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ buffer,
+ metadata_builder,
+ finished: false,
+ }
+ }
+
+ fn list(
+ buffer: &'a mut ValueBuffer,
+ metadata_builder: &'a mut MetadataBuilder,
+ offsets: &'a mut Vec<usize>,
+ saved_parent_buffer_offset: usize,
+ ) -> Self {
+ // The saved_parent_buffer_offset is the buffer size as of when the parent builder was
+ // constructed. The saved_buffer_offset is the buffer size as of now (when a child builder
+ // is created). The variant field_offset entry for this list element is their difference.
+ let saved_buffer_offset = buffer.offset();
+ let saved_offsets_size = offsets.len();
+ offsets.push(saved_buffer_offset - saved_parent_buffer_offset);
+
+ ParentState::List {
+ saved_metadata_builder_dict_size: metadata_builder.num_field_names(),
+ saved_buffer_offset,
+ saved_offsets_size,
+ metadata_builder,
+ buffer,
+ offsets,
+ finished: false,
}
}
+ fn object(
Review Comment:
Good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] ParentState handles finish/rollback for builders [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8185:
URL: https://github.com/apache/arrow-rs/pull/8185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org