[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #2273: Add bytes scanned metric to ParquetExec

[GitBox <gi...@apache.org>]

andygrove commented on code in PR #2273:
URL: https://github.com/apache/arrow-datafusion/pull/2273#discussion_r854317529


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -362,19 +368,27 @@ fn fetch_statistics(
 }
 
 /// A wrapper around the object reader to make it implement `ChunkReader`
-pub struct ChunkObjectReader(pub Arc<dyn ObjectReader>);
+pub struct ChunkObjectReader {
+    /// The underlying object reader
+    pub object_reader: Arc<dyn ObjectReader>,
+    /// Optional counter which will track total number of bytes scanned
+    pub bytes_scanned: Option<metrics::Count>,
+}
 
 impl Length for ChunkObjectReader {
     fn len(&self) -> u64 {
-        self.0.length()
+        self.object_reader.length()
     }
 }
 
 impl ChunkReader for ChunkObjectReader {
     type T = Box<dyn Read + Send + Sync>;
 
     fn get_read(&self, start: u64, length: usize) -> ParquetResult<Self::T> {
-        self.0
+        if let Some(m) = self.bytes_scanned.as_ref() {
+            m.add(length)

Review Comment:
   I'm just curious .. the length here is the number of bytes requested and could potentially be different from the actual number of bytes read?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org