[arrow-datafusion] branch master updated: MINOR: add some comments about intended use of ChunkedStore (#4541)

[tu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git


The following commit(s) were added to refs/heads/master by this push:
     new 4c1f60c6a MINOR: add some comments about intended use of ChunkedStore (#4541)
4c1f60c6a is described below

commit 4c1f60c6a02c810a1f160f32bfd8751c597a1b22
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Wed Dec 7 10:37:30 2022 -0500

    MINOR: add some comments about intended use of ChunkedStore (#4541)
---
 datafusion/core/src/physical_plan/file_format/chunked_store.rs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/datafusion/core/src/physical_plan/file_format/chunked_store.rs b/datafusion/core/src/physical_plan/file_format/chunked_store.rs
index 2af3f1211..49f5e5a93 100644
--- a/datafusion/core/src/physical_plan/file_format/chunked_store.rs
+++ b/datafusion/core/src/physical_plan/file_format/chunked_store.rs
@@ -28,6 +28,13 @@ use std::sync::Arc;
 use tokio::io::{AsyncReadExt, AsyncWrite, BufReader};
 
 /// Wraps a [`ObjectStore`] and makes its get response return chunks
+/// in a controllable manner.
+///
+/// A `ChunkedStore` makes the memory consumption and performance of
+/// the wrapped [`ObjectStore`] worse. It is intended for use within
+/// tests, to control the chunks in the produced output streams. For
+/// example, it is used to verify the delimiting logic in
+/// newline_delimited_stream.
 ///
 /// TODO: Upstream into object_store_rs
 #[derive(Debug)]