You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/16 03:43:51 UTC

[GitHub] [arrow-rs] Ted-Jiang opened a new pull request, #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Ted-Jiang opened a new pull request, #2460:
URL: https://github.com/apache/arrow-rs/pull/2460

   # Which issue does this PR close?
   
   Closes #2459 
   
   # Rationale for this change
    
   <!--
   Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
   Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
   -->
   
   # What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are there any user-facing changes?
   
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r946552003


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -667,12 +742,32 @@ impl<T: Read + Send> PageReader for SerializedPageReader<T> {
 
     fn skip_next_page(&mut self) -> Result<()> {
         match &mut self.buf {
-            SerializedPages::Chunk { .. } => { Err(general_err!("Must set page_offset_index when using skip_next_page in SerializedPageReader.")) }
-            SerializedPages::Pages { offset_index, seen_num_data_pages, .. } => {
+            SerializedPages::Chunk {
+                next_page_header,
+                buf,
+            } => {
+                if let Some(header) = next_page_header.take() {
+                    let to_skip = header.compressed_page_size;
+                    let mut buffer = Vec::with_capacity(to_skip as usize);
+                    buf.read_exact(&mut buffer)?;

Review Comment:
   I try to implement seek for the `PageReader`, there are two types 'File' and 'tokio::bytes' need porcess. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang closed pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang closed pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader
URL: https://github.com/apache/arrow-rs/pull/2460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r946552003


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -667,12 +742,32 @@ impl<T: Read + Send> PageReader for SerializedPageReader<T> {
 
     fn skip_next_page(&mut self) -> Result<()> {
         match &mut self.buf {
-            SerializedPages::Chunk { .. } => { Err(general_err!("Must set page_offset_index when using skip_next_page in SerializedPageReader.")) }
-            SerializedPages::Pages { offset_index, seen_num_data_pages, .. } => {
+            SerializedPages::Chunk {
+                next_page_header,
+                buf,
+            } => {
+                if let Some(header) = next_page_header.take() {
+                    let to_skip = header.compressed_page_size;
+                    let mut buffer = Vec::with_capacity(to_skip as usize);
+                    buf.read_exact(&mut buffer)?;

Review Comment:
   I try to implement seek for the `PageReader`, there are two types 'File' and 'tokio::bytes' need process.
   But the  'tokio::bytes'  not impl `std::io::seek`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r946310285


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -667,12 +742,32 @@ impl<T: Read + Send> PageReader for SerializedPageReader<T> {
 
     fn skip_next_page(&mut self) -> Result<()> {
         match &mut self.buf {
-            SerializedPages::Chunk { .. } => { Err(general_err!("Must set page_offset_index when using skip_next_page in SerializedPageReader.")) }
-            SerializedPages::Pages { offset_index, seen_num_data_pages, .. } => {
+            SerializedPages::Chunk {
+                next_page_header,
+                buf,
+            } => {
+                if let Some(header) = next_page_header.take() {
+                    let to_skip = header.compressed_page_size;
+                    let mut buffer = Vec::with_capacity(to_skip as usize);
+                    buf.read_exact(&mut buffer)?;

Review Comment:
   @tustvold need you help πŸ˜‚. for now `T: Read` should we implement `T: Read + Seek`, is there some magic wayπŸ€”



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948886097


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -480,13 +480,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]

Review Comment:
   you are right! tested in ut
   ```
   #[test]
       fn test() {
           enum  Test {
               Large {
                   page: Option<PageHeader>,
               }
           }
           let test1 = Test::Large { page: None };
   
           println!("{}", std::mem::size_of_val(&test1));
           ()
       }
   ```
   got 320 πŸ˜‚ thanks for your info!
   I should try before askπŸ˜”



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold merged pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] ursabot commented on pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#issuecomment-1219316093

   Benchmark runs are scheduled for baseline = 9f77e4e77db274ced4859036bedbb8619f1252b4 and contender = 12cc067ec9cec24aa1bf43f55e1cd88ebfe6f0a0. 12cc067ec9cec24aa1bf43f55e1cd88ebfe6f0a0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/ad99739eedb246de8bb5d257bedd2f54...341f83ee05eb4f509e2b14e5a6bfee8b/)
   [Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/eb317b7c7b434d3a9109dca22a65ef40...268e7ac067054a0f9d49f02a881f7a8c/)
   [Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8e35ef1e9e5745bc8ceb5251a1b7cae1...0d27940a12474b2bbc11779e2f8a0d23/)
   [Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/f7b6da39b5324d13a76713d13bfa57f3...68abc16c93d7409aa2b97203cdf48bd2/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948775067


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -480,13 +480,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]

Review Comment:
   An option takes up the same space regardless of it is set, they're like C++ union types in that sense. I think you should box the page header probably...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#issuecomment-1219311783

   I'm going to get this in as the fix for master is in #2494 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r946648212


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -667,12 +742,32 @@ impl<T: Read + Send> PageReader for SerializedPageReader<T> {
 
     fn skip_next_page(&mut self) -> Result<()> {
         match &mut self.buf {
-            SerializedPages::Chunk { .. } => { Err(general_err!("Must set page_offset_index when using skip_next_page in SerializedPageReader.")) }
-            SerializedPages::Pages { offset_index, seen_num_data_pages, .. } => {
+            SerializedPages::Chunk {
+                next_page_header,
+                buf,
+            } => {
+                if let Some(header) = next_page_header.take() {
+                    let to_skip = header.compressed_page_size;
+                    let mut buffer = Vec::with_capacity(to_skip as usize);
+                    buf.read_exact(&mut buffer)?;

Review Comment:
   See https://github.com/apache/arrow-rs/issues/2463 for what I'm working on



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948630306


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -480,13 +480,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]

Review Comment:
   without this clippy get
   ```
   error: large size difference between variants
      --> parquet/src/file/serialized_reader.rs:484:5
       |
   484 | /     Values {
   485 | |         /// The current byte offset in the reader
   486 | |         offset: usize,
   487 | |
   ...   |
   492 | |         next_page_header: Option<PageHeader>,
   493 | |     },
       | |_____^ this variant is 336 bytes
       |
       = note: `-D clippy::large-enum-variant` implied by `-D warnings`
   note: and the second-largest variant is 72 bytes:
      --> parquet/src/file/serialized_reader.rs:494:5
   
   ```
   but i think the 336 is wrong because in `PageHeader`, most of the large arg are `option`, clippy can not know when running only one is `some` , so i think remove this check here is reasonable (or wrap 'Box'  i think this will downgrade)πŸ€”
   ```
     pub data_page_header: Option<DataPageHeader>,
     pub index_page_header: Option<IndexPageHeader>,
     pub dictionary_page_header: Option<DictionaryPageHeader>,
     pub data_page_header_v2: Option<DataPageHeaderV2>,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948886097


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -480,13 +480,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]

Review Comment:
   you are right! tested in ut
   ```
   #[test]
       fn test() {
           enum  Test {
               Large {
                   page: Option<PageHeader>,
               }
           }
           let test1 = Test::Large { page: None };
   
           println!("{}", std::mem::size_of_val(&test1));
           ()
       }
   ```
   got 320 πŸ˜‚ thanks for your info!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948876802


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -648,20 +659,60 @@ impl<R: ChunkReader> PageReader for SerializedPageReader<R> {
     }
 
     fn peek_next_page(&mut self) -> Result<Option<PageMetadata>> {
-        match &self.state {
-            SerializedPageReaderState::Values {..} => Err(general_err!("Must set page_offset_index when using peek_next_page in SerializedPageReader.")),
-            SerializedPageReaderState::Pages { page_locations, dictionary_page, total_rows } => {
+        match &mut self.state {
+            SerializedPageReaderState::Values {
+                offset,
+                remaining_bytes,
+                next_page_header,
+            } => {
+                loop {
+                    if *remaining_bytes == 0 {
+                        return Ok(None);
+                    }
+                    return if let Some(header) = next_page_header.take() {

Review Comment:
   Yes, the only way set it none. Is after skip_next_page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] Ted-Jiang commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948635496


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -480,13 +480,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]

Review Comment:
   @tustvold PTAL😊



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r948776505


##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -119,6 +119,7 @@ impl RowSelection {
     }
 
     /// Given an offset index, return the offset ranges for all data pages selected by `self`
+    #[allow(unused)]

Review Comment:
   #[cfg(any(test, feature = "async"))]
   
   Or something



##########
parquet/src/file/serialized_reader.rs:
##########
@@ -648,20 +659,60 @@ impl<R: ChunkReader> PageReader for SerializedPageReader<R> {
     }
 
     fn peek_next_page(&mut self) -> Result<Option<PageMetadata>> {
-        match &self.state {
-            SerializedPageReaderState::Values {..} => Err(general_err!("Must set page_offset_index when using peek_next_page in SerializedPageReader.")),
-            SerializedPageReaderState::Pages { page_locations, dictionary_page, total_rows } => {
+        match &mut self.state {
+            SerializedPageReaderState::Values {
+                offset,
+                remaining_bytes,
+                next_page_header,
+            } => {
+                loop {
+                    if *remaining_bytes == 0 {
+                        return Ok(None);
+                    }
+                    return if let Some(header) = next_page_header.take() {

Review Comment:
   I think this probably shouldn't take but just be as_ref?



##########
parquet/src/file/serialized_reader.rs:
##########
@@ -484,13 +484,17 @@ pub(crate) fn decode_page(
     Ok(result)
 }
 
+#[allow(clippy::large_enum_variant)]
 enum SerializedPageReaderState {
     Values {
         /// The current byte offset in the reader
         offset: usize,
 
         /// The length of the chunk in bytes
         remaining_bytes: usize,
+
+        // If the next page header has already been "peeked", we will cache it and it`s length here
+        next_page_header: Option<PageHeader>,

Review Comment:
   ```suggestion
           next_page_header: Option<Box<PageHeader>>,
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2460: Support skip_page missing OffsetIndex Fallback in SerializedPageReader

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2460:
URL: https://github.com/apache/arrow-rs/pull/2460#discussion_r946491287


##########
parquet/src/file/serialized_reader.rs:
##########
@@ -667,12 +742,32 @@ impl<T: Read + Send> PageReader for SerializedPageReader<T> {
 
     fn skip_next_page(&mut self) -> Result<()> {
         match &mut self.buf {
-            SerializedPages::Chunk { .. } => { Err(general_err!("Must set page_offset_index when using skip_next_page in SerializedPageReader.")) }
-            SerializedPages::Pages { offset_index, seen_num_data_pages, .. } => {
+            SerializedPages::Chunk {
+                next_page_header,
+                buf,
+            } => {
+                if let Some(header) = next_page_header.take() {
+                    let to_skip = header.compressed_page_size;
+                    let mut buffer = Vec::with_capacity(to_skip as usize);
+                    buf.read_exact(&mut buffer)?;

Review Comment:
   I'll have a play and see what I can come up with



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org