[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #6526: Add support for appending data to external tables - CSV

["tustvold (via GitHub)" <gi...@apache.org>]

tustvold commented on code in PR #6526:
URL: https://github.com/apache/arrow-datafusion/pull/6526#discussion_r1216616214


##########
datafusion/core/src/datasource/file_format/mod.rs:
##########
@@ -87,6 +98,277 @@ pub trait FileFormat: Send + Sync + fmt::Debug {
         conf: FileScanConfig,
         filters: Option<&Arc<dyn PhysicalExpr>>,
     ) -> Result<Arc<dyn ExecutionPlan>>;
+
+    /// Take a list of files and the configuration to convert it to the
+    /// appropriate writer executor according to this file format.
+    async fn create_writer_physical_plan(
+        &self,
+        _input: Arc<dyn ExecutionPlan>,
+        _state: &SessionState,
+        _conf: FileSinkConfig,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        let msg = "Writer not implemented for this format".to_owned();
+        Err(DataFusionError::NotImplemented(msg))
+    }
+}
+
+/// `AsyncPutWriter` is an object that facilitates asynchronous writing to object stores.
+/// It is specifically designed for the `object_store` crate's `put` method and sends
+/// whole bytes at once when the buffer is flushed.
+pub struct AsyncPutWriter {

Review Comment:
   I'm confused to see this here, we already provide AsynWrite from ObjectStore, so I'm not sure why we are re-implementing the buffering here? Am i missing something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org