You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/29 03:49:40 UTC

[GitHub] [arrow-rs] bjchambers commented on a change in pull request #1246: Fix NullArrayReader (#1245)

bjchambers commented on a change in pull request #1246:
URL: https://github.com/apache/arrow-rs/pull/1246#discussion_r794993876



##########
File path: parquet/src/arrow/array_reader.rs
##########
@@ -214,6 +214,10 @@ where
         // save definition and repetition buffers
         self.def_levels_buffer = self.record_reader.consume_def_levels()?;
         self.rep_levels_buffer = self.record_reader.consume_rep_levels()?;
+
+        // Must consume bitmap buffer
+        self.record_reader.consume_bitmap_buffer()?;

Review comment:
       One place it comes up is where you have multiple Parquet files representing a "table", with one or more columns being relatively sparse. If you have a file dropped every hour, then it may be that some of the files have a null column while others have a few values.
   
   It doesn't seem likely it would be the bottleneck (the other columns in the file probably would be), but that's at least how it's come up for us.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org