You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2024/10/09 16:40:05 UTC

[PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

adriangb opened a new pull request, #6534:
URL: https://github.com/apache/arrow-rs/pull/6534

   Closes https://github.com/apache/arrow-rs/issues/6529


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1795174120


##########
arrow-buffer/src/builder/null.rs:
##########
@@ -95,6 +95,17 @@ impl NullBufferBuilder {
         self.bitmap_builder.as_mut().unwrap().append_n(n, false);
     }
 
+    /// Appends `n` `true`s into the builder
+    /// to indicate that these `n` items are not null.
+    #[inline]
+    pub fn append_n_non_null(&mut self, n: usize) {

Review Comment:
   I believe this method already exists a few lines up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1795803161


##########
arrow-array/src/builder/primitive_builder.rs:
##########
@@ -202,6 +202,13 @@ impl<T: ArrowPrimitiveType> PrimitiveBuilder<T> {
         self.values_builder.append(v);
     }
 
+    /// Appends a value of type `T` into the builder `n` times
+    #[inline]
+    pub fn append_value_repeated(&mut self, v: T::Native, n: usize) {

Review Comment:
   It is kind of sad that the naming of this can't be consistent, but I don't really have any better suggestions. Perhaps `append_value_n`?? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1794318436


##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -208,7 +208,21 @@ where
     /// value is appended to the values array.
     ///
     /// Returns an error if the new index would overflow the key type.
+    #[inline]
     pub fn append(&mut self, value: impl AsRef<T::Native>) -> Result<K::Native, ArrowError> {
+        self.append_many(value, 1)

Review Comment:
   Will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#issuecomment-2405006476

   Merge conflicts fixed and API removed!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#issuecomment-2403560355

   @tustvold inspired the the existence of https://docs.rs/arrow-array/latest/arrow_array/builder/struct.PrimitiveBuilder.html#method.append_nulls I added methods such that we can now push the repetition all the way down into the buffers, which I think is as efficient as it's going to get.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1794316451


##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -208,7 +208,21 @@ where
     /// value is appended to the values array.
     ///
     /// Returns an error if the new index would overflow the key type.
+    #[inline]

Review Comment:
   I'm happy to remove it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1794280364


##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -208,7 +208,21 @@ where
     /// value is appended to the values array.
     ///
     /// Returns an error if the new index would overflow the key type.
+    #[inline]

Review Comment:
   What is the motivation for this, is there some benchmark you've used? Given how problematic dictionary codegen already is, because everything is generated for each key-value combination, I'm rather wary of this



##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -208,7 +208,21 @@ where
     /// value is appended to the values array.
     ///
     /// Returns an error if the new index would overflow the key type.
+    #[inline]
     pub fn append(&mut self, value: impl AsRef<T::Native>) -> Result<K::Native, ArrowError> {
+        self.append_many(value, 1)

Review Comment:
   Perhaps we could instead extract the entry logic into a private member function, I'd be less concerned about potential performance regressions that way, LLVM does funny things with integer loops



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1794285686


##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -267,6 +294,23 @@ where
         };
     }
 
+    /// Append an `Option` value into the builder repeatedly `count` times.
+    /// This is the same as `append_option` but allows to append the same value multiple times without doing multiple lookups.
+    ///
+    /// # Panics
+    ///
+    /// Panics if the resulting length of the dictionary values array would exceed `T::Native::MAX`
+    pub fn append_option_many(&mut self, value: Option<impl AsRef<T::Native>>, count: usize) {
+        match value {
+            None => {
+                for _ in 0..count {

Review Comment:
   https://docs.rs/arrow-array/latest/arrow_array/builder/struct.PrimitiveBuilder.html#method.append_nulls will be significantly faster



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#discussion_r1795804771


##########
arrow-array/src/builder/primitive_builder.rs:
##########
@@ -202,6 +202,13 @@ impl<T: ArrowPrimitiveType> PrimitiveBuilder<T> {
         self.values_builder.append(v);
     }
 
+    /// Appends a value of type `T` into the builder `n` times
+    #[inline]
+    pub fn append_value_repeated(&mut self, v: T::Native, n: usize) {

Review Comment:
   Yeah I noticed the names are getting a bit messy and mixed up. I'm happy to go with that suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add append_many to dictionary arrays to allow adding repeated values [arrow-rs]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #6534:
URL: https://github.com/apache/arrow-rs/pull/6534#issuecomment-2405797549

   I never addressed this:
   > I wonder if you've given thought to instead providing a way to manually append keys and manually add/lookup values in the dictionary?
   I think it's possible to want those APIs, but in my case I would have turned around and done exactly this. Plus I feel like that exposes more internal bits?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org