You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/20 18:59:06 UTC

[GitHub] [arrow-datafusion] thinkharderdev commented on a diff in pull request #2273: Add bytes scanned metric to ParquetExec

thinkharderdev commented on code in PR #2273:
URL: https://github.com/apache/arrow-datafusion/pull/2273#discussion_r854461112


##########
datafusion/core/src/datasource/file_format/parquet.rs:
##########
@@ -362,19 +368,27 @@ fn fetch_statistics(
 }
 
 /// A wrapper around the object reader to make it implement `ChunkReader`
-pub struct ChunkObjectReader(pub Arc<dyn ObjectReader>);
+pub struct ChunkObjectReader {
+    /// The underlying object reader
+    pub object_reader: Arc<dyn ObjectReader>,
+    /// Optional counter which will track total number of bytes scanned
+    pub bytes_scanned: Option<metrics::Count>,
+}
 
 impl Length for ChunkObjectReader {
     fn len(&self) -> u64 {
-        self.0.length()
+        self.object_reader.length()
     }
 }
 
 impl ChunkReader for ChunkObjectReader {
     type T = Box<dyn Read + Send + Sync>;
 
     fn get_read(&self, start: u64, length: usize) -> ParquetResult<Self::T> {
-        self.0
+        if let Some(m) = self.bytes_scanned.as_ref() {
+            m.add(length)

Review Comment:
   I don't think this interface actually specifies so it would really be up to the specific `ObjectStore` implementation. I know that that is the way the `LocalFileSystem` works and I believe that is the way the AWS S3 API works with file ranges. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org