You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2025/08/16 20:41:00 UTC

[PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

rok opened a new pull request, #8162:
URL: https://github.com/apache/arrow-rs/pull/8162

   - Closes #8115.
   
   # Rationale for this change
   
   See #8115.
   
   # What changes are included in this PR?
   
   TBD.
   
   # Are these changes tested?
   
   This includes a datafusion inspired test for concurrent writting accross columns and row groups.
   
   # Are there any user-facing changes?
   
   TBD.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2361511303


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].

Review Comment:
   > I spent quite a while trying to figure out why we can't just use get_column_writers.
   
   Sorry, I realise now that I didn't make this very clear in my original issue (#7359).
   
   One other factor is that we couldn't just use the `WriterProperties` passed to `get_column_writers` to internally create a new `FileEncryptor`. When a `FileEncryptor` is [created](https://github.com/apache/arrow-rs/blob/30a779878a617116afe73e48ac05abfff0040c6f/parquet/src/file/writer.rs#L187) for the `SerializedFileWriter`, it [generates random AAD](https://github.com/apache/arrow-rs/blob/11158ba8cfc471354ed14d04c88c9c631143459a/parquet/src/encryption/encrypt.rs#L302) (additional authentication data), and this AAD has to be the same for all encrypted modules in the file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2360358117


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -906,6 +918,12 @@ impl ArrowRowGroupWriterFactory {
         let writers = get_column_writers(&self.schema, &self.props, &self.arrow_schema)?;
         Ok(ArrowRowGroupWriter::new(writers, &self.arrow_schema))
     }
+
+    /// Create column writers for a new row group.
+    pub fn create_column_writers(&self, row_group_index: usize) -> Result<Vec<ArrowColumnWriter>> {

Review Comment:
   So I am not sure making `ArrowRowGroupWriter` public gets us much of anything, and it would not allow per-column parallel encoding
   
   One benefit of getting the column writers individually, is that then the columns can be encoded in parallel. The `ArrowRowGroupWriter` can only write RowGroups in parallel. 
   
   I looked at `ArrowRowGroupWriter` a bit more, and the only substantial thing it  does is call a loop with [compute_leaves](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/fn.compute_leaves.html) which is already public. 
   
   
   https://github.com/apache/arrow-rs/blob/bac36900826e411564231b89e3eb544ea9082cab/parquet/src/arrow/arrow_writer/mod.rs#L831-L835



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3311995641

   Thank you @rok @lilianm and @adamreeve 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3214743955

   @alamb @adamreeve I think this will make for a better multithreading API.
   Unfortunately this misses the release window and I assume we now have to deprecate a couple of `ArrowWriter` methods. Do we have a policy for this @alamb ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2296830308


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -527,28 +770,17 @@ async fn test_multi_threaded_encrypted_writing() {
 
     // Create a temporary file to write the encrypted data
     let temp_file = tempfile::tempfile().unwrap();
-    let mut writer = ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props).unwrap();
+    let mut writer = ArrowWriter::try_new(&temp_file, schema.clone(), props).unwrap();
 
     // LOW-LEVEL API: Use low level API to write into a file using multiple threads
 
     // Get column writers
+    #[allow(deprecated)]

Review Comment:
   If we're deprecating the old approach, rather than using `allow(deprecated)`, I think it would be best to update this test to use the new approach instead. Or maybe keep it but also add a new test that uses the new approach. That way, we have a test to confirm the new API works with this simpler parallelization strategy and won't have a gap in tested functionality if/when the deprecated methods are removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3302377232

   @alamb the deprecation warning comes from `pub get_column_writers` and `pub append_row_group` being added to `AsyncArrowWriter` [since last release](https://github.com/apache/arrow-rs/pull/8262). These (append_row_group, append_row_group) were added to `ArrowWriter` just prior the last release, but turned out to not be great API for [downstream](https://github.com/apache/datafusion/pull/16738) writing due to locking issues so we decided to deprecate them to as they were not needed and to reduce `pub` area as [suggested here](https://github.com/apache/arrow-rs/pull/7818#discussion_r2201592759).
   We would suggest removing `pub get_column_writers` and `pub append_row_group` from both `ArrowWriter` and `AsyncArrowWriter` before the next release and this PR reflects that.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2360321979


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].

Review Comment:
   https://github.com/apache/arrow-rs/pull/8162#pullrequestreview-3239077577 is my suggestion , basically TLDR is to make `ArrowRowGroupWriterFactory` constructor public and deprecate `get_column_writers` 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3308624389

   > Sorry for the late response @alamb and thanks for the added changes. I think this is good as is and can be included in the release.
   
   This was my fault for not reviewing this PR before in more detail. 
   
   I leave this open until tomorrow to give people a chance to respond, and if not I'll merge it in and make the RC


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3293945261

   Thanks for the review @alamb!
   It seems we missed a codepath. I'll open a PR tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3303838379

   I am getting ready to create the 56.2.0 release candidate. Shall we try and get this one in, or can it wait for 57.0.0 in October?
    - https://github.com/apache/arrow-rs/pull/8372


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3302408048

   I don't have the mixed [sync/async-write test](https://github.com/apache/arrow-rs/pull/8162/commits/c4e38db434b8e7d2675ead399eaf48cef1926b86) quite right yet, but I wonder if we really need it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2313399562


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -493,8 +503,341 @@ async fn read_and_roundtrip_to_encrypted_file_async(
     verify_encryption_test_file_read_async(&mut file, decryption_properties).await
 }
 
+// Type aliases for multithreaded file writing tests
+type ColSender = Sender<ArrowLeafColumn>;
+type ColumnWriterTask = JoinHandle<Result<ArrowColumnWriter, ParquetError>>;
+type RBStreamSerializeResult = Result<(Vec<ArrowColumnChunk>, usize), ParquetError>;
+
+async fn send_arrays_to_column_writers(
+    col_array_channels: &[ColSender],
+    rb: &RecordBatch,
+    schema: &Arc<Schema>,
+) -> Result<(), ParquetError> {
+    // Each leaf column has its own channel, increment next_channel for each leaf column sent.
+    let mut next_channel = 0;
+    for (array, field) in rb.columns().iter().zip(schema.fields()) {
+        for c in compute_leaves(field, array)? {
+            if col_array_channels[next_channel].send(c).await.is_err() {
+                return Ok(());
+            }
+            next_channel += 1;
+        }
+    }
+    Ok(())
+}
+
+/// Spawns a tokio task which joins the parallel column writer tasks,
+/// and finalizes the row group
+fn spawn_rg_join_and_finalize_task(
+    column_writer_tasks: Vec<ColumnWriterTask>,
+    rg_rows: usize,
+) -> JoinHandle<RBStreamSerializeResult> {
+    tokio::task::spawn(async move {
+        let num_cols = column_writer_tasks.len();
+        let mut finalized_rg = Vec::with_capacity(num_cols);
+        for task in column_writer_tasks.into_iter() {
+            let writer = task
+                .await
+                .map_err(|e| ParquetError::General(e.to_string()))??;
+            finalized_rg.push(writer.close()?);
+        }
+        Ok((finalized_rg, rg_rows))
+    })
+}
+
+fn spawn_parquet_parallel_serialization_task(
+    writer_factory: ArrowRowGroupWriterFactory,
+    mut data: Receiver<RecordBatch>,
+    serialize_tx: Sender<JoinHandle<RBStreamSerializeResult>>,
+    schema: Arc<Schema>,
+) -> JoinHandle<Result<(), ParquetError>> {
+    tokio::spawn(async move {
+        let max_buffer_rb = 10;
+        let max_row_group_rows = 10;
+        let mut row_group_index = 0;
+
+        let column_writers = writer_factory.create_column_writers(row_group_index)?;
+
+        let (mut col_writer_tasks, mut col_array_channels) =
+            spawn_column_parallel_row_group_writer(column_writers, max_buffer_rb)?;
+
+        let mut current_rg_rows = 0;
+
+        while let Some(mut rb) = data.recv().await {
+            // This loop allows the "else" block to repeatedly split the RecordBatch to handle the case
+            // when max_row_group_rows < execution.batch_size as an alternative to a recursive async
+            // function.
+            loop {
+                if current_rg_rows + rb.num_rows() < max_row_group_rows {
+                    send_arrays_to_column_writers(&col_array_channels, &rb, &schema).await?;
+                    current_rg_rows += rb.num_rows();
+                    break;
+                } else {
+                    let rows_left = max_row_group_rows - current_rg_rows;
+                    let rb_split = rb.slice(0, rows_left);
+                    send_arrays_to_column_writers(&col_array_channels, &rb_split, &schema).await?;
+
+                    // Signal the parallel column writers that the RowGroup is done, join and finalize RowGroup
+                    // on a separate task, so that we can immediately start on the next RG before waiting
+                    // for the current one to finish.
+                    drop(col_array_channels);
+
+                    let finalize_rg_task =
+                        spawn_rg_join_and_finalize_task(col_writer_tasks, max_row_group_rows);
+
+                    // Do not surface error from closed channel (means something
+                    // else hit an error, and the plan is shutting down).
+                    if serialize_tx.send(finalize_rg_task).await.is_err() {
+                        return Ok(());
+                    }
+
+                    current_rg_rows = 0;
+                    rb = rb.slice(rows_left, rb.num_rows() - rows_left);
+
+                    row_group_index += 1;
+                    let column_writers = writer_factory.create_column_writers(row_group_index)?;
+                    (col_writer_tasks, col_array_channels) =
+                        spawn_column_parallel_row_group_writer(column_writers, 100)?;
+                }
+            }
+        }
+
+        drop(col_array_channels);
+        // Handle leftover rows as final rowgroup, which may be smaller than max_row_group_rows
+        if current_rg_rows > 0 {
+            let finalize_rg_task =
+                spawn_rg_join_and_finalize_task(col_writer_tasks, current_rg_rows);
+
+            // Do not surface error from closed channel (means something
+            // else hit an error, and the plan is shutting down).
+            if serialize_tx.send(finalize_rg_task).await.is_err() {
+                return Ok(());
+            }
+        }
+
+        Ok(())
+    })
+}
+
+fn spawn_column_parallel_row_group_writer(
+    col_writers: Vec<ArrowColumnWriter>,
+    max_buffer_size: usize,
+) -> Result<(Vec<ColumnWriterTask>, Vec<ColSender>), ParquetError> {
+    let num_columns = col_writers.len();
+
+    let mut col_writer_tasks = Vec::with_capacity(num_columns);
+    let mut col_array_channels = Vec::with_capacity(num_columns);
+    for mut col_writer in col_writers.into_iter() {
+        let (send_array, mut receive_array) =
+            tokio::sync::mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
+        col_array_channels.push(send_array);
+        let handle = tokio::spawn(async move {
+            while let Some(col) = receive_array.recv().await {
+                col_writer.write(&col)?;
+            }
+            Ok(col_writer)
+        });
+        col_writer_tasks.push(handle);
+    }
+    Ok((col_writer_tasks, col_array_channels))
+}
+
+/// Consume RowGroups serialized by other parallel tasks and concatenate them
+/// to the final parquet file
+async fn concatenate_parallel_row_groups<W: Write + Send>(
+    mut parquet_writer: SerializedFileWriter<W>,
+    mut serialize_rx: Receiver<JoinHandle<RBStreamSerializeResult>>,
+) -> Result<FileMetaData, ParquetError> {
+    while let Some(task) = serialize_rx.recv().await {
+        let result = task.await;
+        let mut rg_out = parquet_writer.next_row_group()?;
+        let (serialized_columns, _cnt) =
+            result.map_err(|e| ParquetError::General(e.to_string()))??;
+
+        for column_chunk in serialized_columns {
+            column_chunk.append_to_row_group(&mut rg_out)?;
+        }
+        rg_out.close()?;
+    }
+
+    let file_metadata = parquet_writer.close()?;
+    Ok(file_metadata)
+}
+
+// This test is based on DataFusion's ParquetSink. Motivation is to test
+// concurrent writing of encrypted data over multiple row groups using the low-level API.
+#[tokio::test]
+async fn test_concurrent_encrypted_writing_over_multiple_row_groups() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema();
+
+    // Create a channel to send RecordBatches to the writer and send row groups
+    let (record_batch_tx, data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let arrow_writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (writer, row_group_writer_factory) = arrow_writer.into_serialized_writer().unwrap();
+    let max_row_groups = 1;
+
+    let (serialize_tx, serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(max_row_groups);
+
+    let launch_serialization_task = spawn_parquet_parallel_serialization_task(
+        row_group_writer_factory,
+        data,
+        serialize_tx,
+        schema.clone(),
+    );
+
+    let _file_metadata = concatenate_parallel_row_groups(writer, serialize_rx)
+        .await
+        .unwrap();
+
+    data_generator.await.unwrap();
+    launch_serialization_task.await.unwrap().unwrap();
+
+    // Check that the file was written correctly
+    let (read_record_batches, read_metadata) =
+        read_encrypted_file(&temp_file, decryption_properties.clone()).unwrap();
+
+    assert_eq!(read_metadata.metadata().file_metadata().num_rows(), 50);
+    verify_encryption_test_data(read_record_batches, read_metadata.metadata());
+}
+
 #[tokio::test]
-async fn test_multi_threaded_encrypted_writing() {
+async fn test_multi_threaded_encrypted_writing_replace_deprecated_api() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema().clone();
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (mut serialized_file_writer, row_group_writer_factory) =
+        writer.into_serialized_writer().unwrap();
+
+    let (serialize_tx, mut serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(1);
+
+    // Create a channel to send RecordBatches to the writer and send row batches
+    let (record_batch_tx, mut data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    // Get column writers
+    // This is instead of let col_writers = writer.get_column_writers().unwrap();

Review Comment:
   Ok, removing this comment too:
   ```rust
   // This is instead of arrow_writer.append_row_group(arrow_column_chunks)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2312642029


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -493,8 +503,341 @@ async fn read_and_roundtrip_to_encrypted_file_async(
     verify_encryption_test_file_read_async(&mut file, decryption_properties).await
 }
 
+// Type aliases for multithreaded file writing tests
+type ColSender = Sender<ArrowLeafColumn>;
+type ColumnWriterTask = JoinHandle<Result<ArrowColumnWriter, ParquetError>>;
+type RBStreamSerializeResult = Result<(Vec<ArrowColumnChunk>, usize), ParquetError>;
+
+async fn send_arrays_to_column_writers(
+    col_array_channels: &[ColSender],
+    rb: &RecordBatch,
+    schema: &Arc<Schema>,
+) -> Result<(), ParquetError> {
+    // Each leaf column has its own channel, increment next_channel for each leaf column sent.
+    let mut next_channel = 0;
+    for (array, field) in rb.columns().iter().zip(schema.fields()) {
+        for c in compute_leaves(field, array)? {
+            if col_array_channels[next_channel].send(c).await.is_err() {
+                return Ok(());
+            }
+            next_channel += 1;
+        }
+    }
+    Ok(())
+}
+
+/// Spawns a tokio task which joins the parallel column writer tasks,
+/// and finalizes the row group
+fn spawn_rg_join_and_finalize_task(
+    column_writer_tasks: Vec<ColumnWriterTask>,
+    rg_rows: usize,
+) -> JoinHandle<RBStreamSerializeResult> {
+    tokio::task::spawn(async move {
+        let num_cols = column_writer_tasks.len();
+        let mut finalized_rg = Vec::with_capacity(num_cols);
+        for task in column_writer_tasks.into_iter() {
+            let writer = task
+                .await
+                .map_err(|e| ParquetError::General(e.to_string()))??;
+            finalized_rg.push(writer.close()?);
+        }
+        Ok((finalized_rg, rg_rows))
+    })
+}
+
+fn spawn_parquet_parallel_serialization_task(
+    writer_factory: ArrowRowGroupWriterFactory,
+    mut data: Receiver<RecordBatch>,
+    serialize_tx: Sender<JoinHandle<RBStreamSerializeResult>>,
+    schema: Arc<Schema>,
+) -> JoinHandle<Result<(), ParquetError>> {
+    tokio::spawn(async move {
+        let max_buffer_rb = 10;
+        let max_row_group_rows = 10;
+        let mut row_group_index = 0;
+
+        let column_writers = writer_factory.create_column_writers(row_group_index)?;
+
+        let (mut col_writer_tasks, mut col_array_channels) =
+            spawn_column_parallel_row_group_writer(column_writers, max_buffer_rb)?;
+
+        let mut current_rg_rows = 0;
+
+        while let Some(mut rb) = data.recv().await {
+            // This loop allows the "else" block to repeatedly split the RecordBatch to handle the case
+            // when max_row_group_rows < execution.batch_size as an alternative to a recursive async
+            // function.
+            loop {
+                if current_rg_rows + rb.num_rows() < max_row_group_rows {
+                    send_arrays_to_column_writers(&col_array_channels, &rb, &schema).await?;
+                    current_rg_rows += rb.num_rows();
+                    break;
+                } else {
+                    let rows_left = max_row_group_rows - current_rg_rows;
+                    let rb_split = rb.slice(0, rows_left);
+                    send_arrays_to_column_writers(&col_array_channels, &rb_split, &schema).await?;
+
+                    // Signal the parallel column writers that the RowGroup is done, join and finalize RowGroup
+                    // on a separate task, so that we can immediately start on the next RG before waiting
+                    // for the current one to finish.
+                    drop(col_array_channels);
+
+                    let finalize_rg_task =
+                        spawn_rg_join_and_finalize_task(col_writer_tasks, max_row_group_rows);
+
+                    // Do not surface error from closed channel (means something
+                    // else hit an error, and the plan is shutting down).
+                    if serialize_tx.send(finalize_rg_task).await.is_err() {
+                        return Ok(());
+                    }
+
+                    current_rg_rows = 0;
+                    rb = rb.slice(rows_left, rb.num_rows() - rows_left);
+
+                    row_group_index += 1;
+                    let column_writers = writer_factory.create_column_writers(row_group_index)?;
+                    (col_writer_tasks, col_array_channels) =
+                        spawn_column_parallel_row_group_writer(column_writers, 100)?;
+                }
+            }
+        }
+
+        drop(col_array_channels);
+        // Handle leftover rows as final rowgroup, which may be smaller than max_row_group_rows
+        if current_rg_rows > 0 {
+            let finalize_rg_task =
+                spawn_rg_join_and_finalize_task(col_writer_tasks, current_rg_rows);
+
+            // Do not surface error from closed channel (means something
+            // else hit an error, and the plan is shutting down).
+            if serialize_tx.send(finalize_rg_task).await.is_err() {
+                return Ok(());
+            }
+        }
+
+        Ok(())
+    })
+}
+
+fn spawn_column_parallel_row_group_writer(
+    col_writers: Vec<ArrowColumnWriter>,
+    max_buffer_size: usize,
+) -> Result<(Vec<ColumnWriterTask>, Vec<ColSender>), ParquetError> {
+    let num_columns = col_writers.len();
+
+    let mut col_writer_tasks = Vec::with_capacity(num_columns);
+    let mut col_array_channels = Vec::with_capacity(num_columns);
+    for mut col_writer in col_writers.into_iter() {
+        let (send_array, mut receive_array) =
+            tokio::sync::mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
+        col_array_channels.push(send_array);
+        let handle = tokio::spawn(async move {
+            while let Some(col) = receive_array.recv().await {
+                col_writer.write(&col)?;
+            }
+            Ok(col_writer)
+        });
+        col_writer_tasks.push(handle);
+    }
+    Ok((col_writer_tasks, col_array_channels))
+}
+
+/// Consume RowGroups serialized by other parallel tasks and concatenate them
+/// to the final parquet file
+async fn concatenate_parallel_row_groups<W: Write + Send>(
+    mut parquet_writer: SerializedFileWriter<W>,
+    mut serialize_rx: Receiver<JoinHandle<RBStreamSerializeResult>>,
+) -> Result<FileMetaData, ParquetError> {
+    while let Some(task) = serialize_rx.recv().await {
+        let result = task.await;
+        let mut rg_out = parquet_writer.next_row_group()?;
+        let (serialized_columns, _cnt) =
+            result.map_err(|e| ParquetError::General(e.to_string()))??;
+
+        for column_chunk in serialized_columns {
+            column_chunk.append_to_row_group(&mut rg_out)?;
+        }
+        rg_out.close()?;
+    }
+
+    let file_metadata = parquet_writer.close()?;
+    Ok(file_metadata)
+}
+
+// This test is based on DataFusion's ParquetSink. Motivation is to test
+// concurrent writing of encrypted data over multiple row groups using the low-level API.
+#[tokio::test]
+async fn test_concurrent_encrypted_writing_over_multiple_row_groups() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema();
+
+    // Create a channel to send RecordBatches to the writer and send row groups
+    let (record_batch_tx, data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let arrow_writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (writer, row_group_writer_factory) = arrow_writer.into_serialized_writer().unwrap();
+    let max_row_groups = 1;
+
+    let (serialize_tx, serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(max_row_groups);
+
+    let launch_serialization_task = spawn_parquet_parallel_serialization_task(
+        row_group_writer_factory,
+        data,
+        serialize_tx,
+        schema.clone(),
+    );
+
+    let _file_metadata = concatenate_parallel_row_groups(writer, serialize_rx)
+        .await
+        .unwrap();
+
+    data_generator.await.unwrap();
+    launch_serialization_task.await.unwrap().unwrap();
+
+    // Check that the file was written correctly
+    let (read_record_batches, read_metadata) =
+        read_encrypted_file(&temp_file, decryption_properties.clone()).unwrap();
+
+    assert_eq!(read_metadata.metadata().file_metadata().num_rows(), 50);
+    verify_encryption_test_data(read_record_batches, read_metadata.metadata());
+}
+
 #[tokio::test]
-async fn test_multi_threaded_encrypted_writing() {
+async fn test_multi_threaded_encrypted_writing_replace_deprecated_api() {

Review Comment:
   I think the `replace_deprecated_api` isn't necessary and will only confuse readers who don't know all the history.
   
   ```suggestion
   async fn test_multi_threaded_encrypted_writing() {
   ```



##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -493,8 +503,341 @@ async fn read_and_roundtrip_to_encrypted_file_async(
     verify_encryption_test_file_read_async(&mut file, decryption_properties).await
 }
 
+// Type aliases for multithreaded file writing tests
+type ColSender = Sender<ArrowLeafColumn>;
+type ColumnWriterTask = JoinHandle<Result<ArrowColumnWriter, ParquetError>>;
+type RBStreamSerializeResult = Result<(Vec<ArrowColumnChunk>, usize), ParquetError>;
+
+async fn send_arrays_to_column_writers(
+    col_array_channels: &[ColSender],
+    rb: &RecordBatch,
+    schema: &Arc<Schema>,
+) -> Result<(), ParquetError> {
+    // Each leaf column has its own channel, increment next_channel for each leaf column sent.
+    let mut next_channel = 0;
+    for (array, field) in rb.columns().iter().zip(schema.fields()) {
+        for c in compute_leaves(field, array)? {
+            if col_array_channels[next_channel].send(c).await.is_err() {
+                return Ok(());
+            }
+            next_channel += 1;
+        }
+    }
+    Ok(())
+}
+
+/// Spawns a tokio task which joins the parallel column writer tasks,
+/// and finalizes the row group
+fn spawn_rg_join_and_finalize_task(
+    column_writer_tasks: Vec<ColumnWriterTask>,
+    rg_rows: usize,
+) -> JoinHandle<RBStreamSerializeResult> {
+    tokio::task::spawn(async move {
+        let num_cols = column_writer_tasks.len();
+        let mut finalized_rg = Vec::with_capacity(num_cols);
+        for task in column_writer_tasks.into_iter() {
+            let writer = task
+                .await
+                .map_err(|e| ParquetError::General(e.to_string()))??;
+            finalized_rg.push(writer.close()?);
+        }
+        Ok((finalized_rg, rg_rows))
+    })
+}
+
+fn spawn_parquet_parallel_serialization_task(
+    writer_factory: ArrowRowGroupWriterFactory,
+    mut data: Receiver<RecordBatch>,
+    serialize_tx: Sender<JoinHandle<RBStreamSerializeResult>>,
+    schema: Arc<Schema>,
+) -> JoinHandle<Result<(), ParquetError>> {
+    tokio::spawn(async move {
+        let max_buffer_rb = 10;
+        let max_row_group_rows = 10;
+        let mut row_group_index = 0;
+
+        let column_writers = writer_factory.create_column_writers(row_group_index)?;
+
+        let (mut col_writer_tasks, mut col_array_channels) =
+            spawn_column_parallel_row_group_writer(column_writers, max_buffer_rb)?;
+
+        let mut current_rg_rows = 0;
+
+        while let Some(mut rb) = data.recv().await {
+            // This loop allows the "else" block to repeatedly split the RecordBatch to handle the case
+            // when max_row_group_rows < execution.batch_size as an alternative to a recursive async
+            // function.
+            loop {
+                if current_rg_rows + rb.num_rows() < max_row_group_rows {
+                    send_arrays_to_column_writers(&col_array_channels, &rb, &schema).await?;
+                    current_rg_rows += rb.num_rows();
+                    break;
+                } else {
+                    let rows_left = max_row_group_rows - current_rg_rows;
+                    let rb_split = rb.slice(0, rows_left);
+                    send_arrays_to_column_writers(&col_array_channels, &rb_split, &schema).await?;
+
+                    // Signal the parallel column writers that the RowGroup is done, join and finalize RowGroup
+                    // on a separate task, so that we can immediately start on the next RG before waiting
+                    // for the current one to finish.
+                    drop(col_array_channels);
+
+                    let finalize_rg_task =
+                        spawn_rg_join_and_finalize_task(col_writer_tasks, max_row_group_rows);
+
+                    // Do not surface error from closed channel (means something
+                    // else hit an error, and the plan is shutting down).
+                    if serialize_tx.send(finalize_rg_task).await.is_err() {
+                        return Ok(());
+                    }
+
+                    current_rg_rows = 0;
+                    rb = rb.slice(rows_left, rb.num_rows() - rows_left);
+
+                    row_group_index += 1;
+                    let column_writers = writer_factory.create_column_writers(row_group_index)?;
+                    (col_writer_tasks, col_array_channels) =
+                        spawn_column_parallel_row_group_writer(column_writers, 100)?;
+                }
+            }
+        }
+
+        drop(col_array_channels);
+        // Handle leftover rows as final rowgroup, which may be smaller than max_row_group_rows
+        if current_rg_rows > 0 {
+            let finalize_rg_task =
+                spawn_rg_join_and_finalize_task(col_writer_tasks, current_rg_rows);
+
+            // Do not surface error from closed channel (means something
+            // else hit an error, and the plan is shutting down).
+            if serialize_tx.send(finalize_rg_task).await.is_err() {
+                return Ok(());
+            }
+        }
+
+        Ok(())
+    })
+}
+
+fn spawn_column_parallel_row_group_writer(
+    col_writers: Vec<ArrowColumnWriter>,
+    max_buffer_size: usize,
+) -> Result<(Vec<ColumnWriterTask>, Vec<ColSender>), ParquetError> {
+    let num_columns = col_writers.len();
+
+    let mut col_writer_tasks = Vec::with_capacity(num_columns);
+    let mut col_array_channels = Vec::with_capacity(num_columns);
+    for mut col_writer in col_writers.into_iter() {
+        let (send_array, mut receive_array) =
+            tokio::sync::mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
+        col_array_channels.push(send_array);
+        let handle = tokio::spawn(async move {
+            while let Some(col) = receive_array.recv().await {
+                col_writer.write(&col)?;
+            }
+            Ok(col_writer)
+        });
+        col_writer_tasks.push(handle);
+    }
+    Ok((col_writer_tasks, col_array_channels))
+}
+
+/// Consume RowGroups serialized by other parallel tasks and concatenate them
+/// to the final parquet file
+async fn concatenate_parallel_row_groups<W: Write + Send>(
+    mut parquet_writer: SerializedFileWriter<W>,
+    mut serialize_rx: Receiver<JoinHandle<RBStreamSerializeResult>>,
+) -> Result<FileMetaData, ParquetError> {
+    while let Some(task) = serialize_rx.recv().await {
+        let result = task.await;
+        let mut rg_out = parquet_writer.next_row_group()?;
+        let (serialized_columns, _cnt) =
+            result.map_err(|e| ParquetError::General(e.to_string()))??;
+
+        for column_chunk in serialized_columns {
+            column_chunk.append_to_row_group(&mut rg_out)?;
+        }
+        rg_out.close()?;
+    }
+
+    let file_metadata = parquet_writer.close()?;
+    Ok(file_metadata)
+}
+
+// This test is based on DataFusion's ParquetSink. Motivation is to test
+// concurrent writing of encrypted data over multiple row groups using the low-level API.
+#[tokio::test]
+async fn test_concurrent_encrypted_writing_over_multiple_row_groups() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema();
+
+    // Create a channel to send RecordBatches to the writer and send row groups
+    let (record_batch_tx, data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let arrow_writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (writer, row_group_writer_factory) = arrow_writer.into_serialized_writer().unwrap();
+    let max_row_groups = 1;
+
+    let (serialize_tx, serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(max_row_groups);
+
+    let launch_serialization_task = spawn_parquet_parallel_serialization_task(
+        row_group_writer_factory,
+        data,
+        serialize_tx,
+        schema.clone(),
+    );
+
+    let _file_metadata = concatenate_parallel_row_groups(writer, serialize_rx)
+        .await
+        .unwrap();
+
+    data_generator.await.unwrap();
+    launch_serialization_task.await.unwrap().unwrap();
+
+    // Check that the file was written correctly
+    let (read_record_batches, read_metadata) =
+        read_encrypted_file(&temp_file, decryption_properties.clone()).unwrap();
+
+    assert_eq!(read_metadata.metadata().file_metadata().num_rows(), 50);
+    verify_encryption_test_data(read_record_batches, read_metadata.metadata());
+}
+
 #[tokio::test]
-async fn test_multi_threaded_encrypted_writing() {
+async fn test_multi_threaded_encrypted_writing_replace_deprecated_api() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema().clone();
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (mut serialized_file_writer, row_group_writer_factory) =
+        writer.into_serialized_writer().unwrap();
+
+    let (serialize_tx, mut serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(1);
+
+    // Create a channel to send RecordBatches to the writer and send row batches
+    let (record_batch_tx, mut data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    // Get column writers
+    // This is instead of let col_writers = writer.get_column_writers().unwrap();

Review Comment:
   I'd also move this line, when the deprecated method is later removed, we don't want to have to remember to remove this comment too.  And users wanting to understand the current state of the code shouldn't be concerned with deprecated methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3293955136

   @alamb I've pushed a couple of ignores just to confirm the codepath. Please don't merge yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2359994359


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].

Review Comment:
   Agreed it's odd. Motivation was to introduce as few new `pub`s as possible. Would be very curious about alternative API shapes.



##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -288,34 +288,17 @@ impl<W: AsyncFileWriter> AsyncArrowWriter<W> {
 
         Ok(())
     }
-
-    /// Create a new row group writer and return its column writers.
-    pub async fn get_column_writers(&mut self) -> Result<Vec<ArrowColumnWriter>> {

Review Comment:
   It is very fresh indeed. I hope we're not spoiling some plan here. We did find the `ArrowRowGroupWriterFactory` route better so I think it should be ok.



##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -349,51 +332,6 @@ mod tests {
         assert_eq!(to_write, read);
     }
 
-    #[tokio::test]

Review Comment:
   Agreed.



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].
+    /// This can be useful to provide more control over how files are written.
+    pub fn into_serialized_writer(

Review Comment:
   Agreed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3303863017

   @alamb it'd be great to get this one in. Let me take a look if I can fix up the `test_async_arrow_group_writer` test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2312642213


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -493,8 +503,341 @@ async fn read_and_roundtrip_to_encrypted_file_async(
     verify_encryption_test_file_read_async(&mut file, decryption_properties).await
 }
 
+// Type aliases for multithreaded file writing tests
+type ColSender = Sender<ArrowLeafColumn>;
+type ColumnWriterTask = JoinHandle<Result<ArrowColumnWriter, ParquetError>>;
+type RBStreamSerializeResult = Result<(Vec<ArrowColumnChunk>, usize), ParquetError>;
+
+async fn send_arrays_to_column_writers(
+    col_array_channels: &[ColSender],
+    rb: &RecordBatch,
+    schema: &Arc<Schema>,
+) -> Result<(), ParquetError> {
+    // Each leaf column has its own channel, increment next_channel for each leaf column sent.
+    let mut next_channel = 0;
+    for (array, field) in rb.columns().iter().zip(schema.fields()) {
+        for c in compute_leaves(field, array)? {
+            if col_array_channels[next_channel].send(c).await.is_err() {
+                return Ok(());
+            }
+            next_channel += 1;
+        }
+    }
+    Ok(())
+}
+
+/// Spawns a tokio task which joins the parallel column writer tasks,
+/// and finalizes the row group
+fn spawn_rg_join_and_finalize_task(
+    column_writer_tasks: Vec<ColumnWriterTask>,
+    rg_rows: usize,
+) -> JoinHandle<RBStreamSerializeResult> {
+    tokio::task::spawn(async move {
+        let num_cols = column_writer_tasks.len();
+        let mut finalized_rg = Vec::with_capacity(num_cols);
+        for task in column_writer_tasks.into_iter() {
+            let writer = task
+                .await
+                .map_err(|e| ParquetError::General(e.to_string()))??;
+            finalized_rg.push(writer.close()?);
+        }
+        Ok((finalized_rg, rg_rows))
+    })
+}
+
+fn spawn_parquet_parallel_serialization_task(
+    writer_factory: ArrowRowGroupWriterFactory,
+    mut data: Receiver<RecordBatch>,
+    serialize_tx: Sender<JoinHandle<RBStreamSerializeResult>>,
+    schema: Arc<Schema>,
+) -> JoinHandle<Result<(), ParquetError>> {
+    tokio::spawn(async move {
+        let max_buffer_rb = 10;
+        let max_row_group_rows = 10;
+        let mut row_group_index = 0;
+
+        let column_writers = writer_factory.create_column_writers(row_group_index)?;
+
+        let (mut col_writer_tasks, mut col_array_channels) =
+            spawn_column_parallel_row_group_writer(column_writers, max_buffer_rb)?;
+
+        let mut current_rg_rows = 0;
+
+        while let Some(mut rb) = data.recv().await {
+            // This loop allows the "else" block to repeatedly split the RecordBatch to handle the case
+            // when max_row_group_rows < execution.batch_size as an alternative to a recursive async
+            // function.
+            loop {
+                if current_rg_rows + rb.num_rows() < max_row_group_rows {
+                    send_arrays_to_column_writers(&col_array_channels, &rb, &schema).await?;
+                    current_rg_rows += rb.num_rows();
+                    break;
+                } else {
+                    let rows_left = max_row_group_rows - current_rg_rows;
+                    let rb_split = rb.slice(0, rows_left);
+                    send_arrays_to_column_writers(&col_array_channels, &rb_split, &schema).await?;
+
+                    // Signal the parallel column writers that the RowGroup is done, join and finalize RowGroup
+                    // on a separate task, so that we can immediately start on the next RG before waiting
+                    // for the current one to finish.
+                    drop(col_array_channels);
+
+                    let finalize_rg_task =
+                        spawn_rg_join_and_finalize_task(col_writer_tasks, max_row_group_rows);
+
+                    // Do not surface error from closed channel (means something
+                    // else hit an error, and the plan is shutting down).
+                    if serialize_tx.send(finalize_rg_task).await.is_err() {
+                        return Ok(());
+                    }
+
+                    current_rg_rows = 0;
+                    rb = rb.slice(rows_left, rb.num_rows() - rows_left);
+
+                    row_group_index += 1;
+                    let column_writers = writer_factory.create_column_writers(row_group_index)?;
+                    (col_writer_tasks, col_array_channels) =
+                        spawn_column_parallel_row_group_writer(column_writers, 100)?;
+                }
+            }
+        }
+
+        drop(col_array_channels);
+        // Handle leftover rows as final rowgroup, which may be smaller than max_row_group_rows
+        if current_rg_rows > 0 {
+            let finalize_rg_task =
+                spawn_rg_join_and_finalize_task(col_writer_tasks, current_rg_rows);
+
+            // Do not surface error from closed channel (means something
+            // else hit an error, and the plan is shutting down).
+            if serialize_tx.send(finalize_rg_task).await.is_err() {
+                return Ok(());
+            }
+        }
+
+        Ok(())
+    })
+}
+
+fn spawn_column_parallel_row_group_writer(
+    col_writers: Vec<ArrowColumnWriter>,
+    max_buffer_size: usize,
+) -> Result<(Vec<ColumnWriterTask>, Vec<ColSender>), ParquetError> {
+    let num_columns = col_writers.len();
+
+    let mut col_writer_tasks = Vec::with_capacity(num_columns);
+    let mut col_array_channels = Vec::with_capacity(num_columns);
+    for mut col_writer in col_writers.into_iter() {
+        let (send_array, mut receive_array) =
+            tokio::sync::mpsc::channel::<ArrowLeafColumn>(max_buffer_size);
+        col_array_channels.push(send_array);
+        let handle = tokio::spawn(async move {
+            while let Some(col) = receive_array.recv().await {
+                col_writer.write(&col)?;
+            }
+            Ok(col_writer)
+        });
+        col_writer_tasks.push(handle);
+    }
+    Ok((col_writer_tasks, col_array_channels))
+}
+
+/// Consume RowGroups serialized by other parallel tasks and concatenate them
+/// to the final parquet file
+async fn concatenate_parallel_row_groups<W: Write + Send>(
+    mut parquet_writer: SerializedFileWriter<W>,
+    mut serialize_rx: Receiver<JoinHandle<RBStreamSerializeResult>>,
+) -> Result<FileMetaData, ParquetError> {
+    while let Some(task) = serialize_rx.recv().await {
+        let result = task.await;
+        let mut rg_out = parquet_writer.next_row_group()?;
+        let (serialized_columns, _cnt) =
+            result.map_err(|e| ParquetError::General(e.to_string()))??;
+
+        for column_chunk in serialized_columns {
+            column_chunk.append_to_row_group(&mut rg_out)?;
+        }
+        rg_out.close()?;
+    }
+
+    let file_metadata = parquet_writer.close()?;
+    Ok(file_metadata)
+}
+
+// This test is based on DataFusion's ParquetSink. Motivation is to test
+// concurrent writing of encrypted data over multiple row groups using the low-level API.
+#[tokio::test]
+async fn test_concurrent_encrypted_writing_over_multiple_row_groups() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema();
+
+    // Create a channel to send RecordBatches to the writer and send row groups
+    let (record_batch_tx, data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let arrow_writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (writer, row_group_writer_factory) = arrow_writer.into_serialized_writer().unwrap();
+    let max_row_groups = 1;
+
+    let (serialize_tx, serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(max_row_groups);
+
+    let launch_serialization_task = spawn_parquet_parallel_serialization_task(
+        row_group_writer_factory,
+        data,
+        serialize_tx,
+        schema.clone(),
+    );
+
+    let _file_metadata = concatenate_parallel_row_groups(writer, serialize_rx)
+        .await
+        .unwrap();
+
+    data_generator.await.unwrap();
+    launch_serialization_task.await.unwrap().unwrap();
+
+    // Check that the file was written correctly
+    let (read_record_batches, read_metadata) =
+        read_encrypted_file(&temp_file, decryption_properties.clone()).unwrap();
+
+    assert_eq!(read_metadata.metadata().file_metadata().num_rows(), 50);
+    verify_encryption_test_data(read_record_batches, read_metadata.metadata());
+}
+
 #[tokio::test]
-async fn test_multi_threaded_encrypted_writing() {
+async fn test_multi_threaded_encrypted_writing_replace_deprecated_api() {
+    // Read example data and set up encryption/decryption properties
+    let testdata = arrow::util::test_util::parquet_test_data();
+    let path = format!("{testdata}/encrypt_columns_and_footer.parquet.encrypted");
+    let file = std::fs::File::open(path).unwrap();
+
+    let file_encryption_properties = FileEncryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+    let decryption_properties = FileDecryptionProperties::builder(b"0123456789012345".into())
+        .with_column_key("double_field", b"1234567890123450".into())
+        .with_column_key("float_field", b"1234567890123451".into())
+        .build()
+        .unwrap();
+
+    let (record_batches, metadata) =
+        read_encrypted_file(&file, decryption_properties.clone()).unwrap();
+    let schema = metadata.schema().clone();
+
+    let props = Some(
+        WriterPropertiesBuilder::default()
+            .with_file_encryption_properties(file_encryption_properties)
+            .build(),
+    );
+
+    // Create a temporary file to write the encrypted data
+    let temp_file = tempfile::tempfile().unwrap();
+    let writer =
+        ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props.clone()).unwrap();
+
+    let (mut serialized_file_writer, row_group_writer_factory) =
+        writer.into_serialized_writer().unwrap();
+
+    let (serialize_tx, mut serialize_rx) =
+        tokio::sync::mpsc::channel::<JoinHandle<RBStreamSerializeResult>>(1);
+
+    // Create a channel to send RecordBatches to the writer and send row batches
+    let (record_batch_tx, mut data) = tokio::sync::mpsc::channel::<RecordBatch>(100);
+    let data_generator = tokio::spawn(async move {
+        for record_batch in record_batches {
+            record_batch_tx.send(record_batch).await.unwrap();
+        }
+    });
+
+    // Get column writers
+    // This is instead of let col_writers = writer.get_column_writers().unwrap();

Review Comment:
   I'd also move this line. When the deprecated method is later removed, we don't want to have to remember to remove this comment too. And users wanting to understand the current state of the code shouldn't be concerned with deprecated methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3304026259

   The motivation for #8262 seems to be making the `AsyncArrowWriter` API match `ArrowWriter`. Is the appropriate fix then to add `into_serialized_writer` to `AsyncArrowWriter` instead of `get_column_writers` and `append_row_group`?
   
   cc @lilianm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3311706761

   > If you'd rather not have the new into_serialized_writer method and require creating the ArrowRowGroupWriterFactory directly, maybe we should take some more time to implement your suggested approach and wait for the next release. But I think into_serialized_writer is convenient and unlikely to cause much extra maintenance burden.
   
   Yeah I agree -- one method is not a big deal if we are going to go with the ArrowRowGroupWriterFactory 
   
   I'll merge this PR and file some follow on tickets shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "lilianm (via GitHub)" <gi...@apache.org>.
lilianm commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2360096024


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -906,6 +918,12 @@ impl ArrowRowGroupWriterFactory {
         let writers = get_column_writers(&self.schema, &self.props, &self.arrow_schema)?;
         Ok(ArrowRowGroupWriter::new(writers, &self.arrow_schema))
     }
+
+    /// Create column writers for a new row group.
+    pub fn create_column_writers(&self, row_group_index: usize) -> Result<Vec<ArrowColumnWriter>> {

Review Comment:
   What do you thinks to set `create_row_group_writer` pub and remove this function and use function `into_writers` add in issue #8260



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3306949112

   > The motivation for #8262 seems to be making the `AsyncArrowWriter` API match `ArrowWriter`. Is the appropriate fix then to add `into_serialized_writer` to `AsyncArrowWriter` instead of `get_column_writers` and `append_row_group`?
   > 
   > cc @lilianm
   
   I am not sure -- what I would love to see is an example like this https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel
   
   That shows how to use whatever APIs we have to write data in parallel
   
   I am sorry I haven't found time to study this PR in detail. @adamreeve it sounds like you have some good ideas and have devoted time here. What would you recommend?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2300547483


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -527,28 +770,17 @@ async fn test_multi_threaded_encrypted_writing() {
 
     // Create a temporary file to write the encrypted data
     let temp_file = tempfile::tempfile().unwrap();
-    let mut writer = ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props).unwrap();
+    let mut writer = ArrowWriter::try_new(&temp_file, schema.clone(), props).unwrap();
 
     // LOW-LEVEL API: Use low level API to write into a file using multiple threads
 
     // Get column writers
+    #[allow(deprecated)]

Review Comment:
   I agree with this suggestion, for the reasons (so well) articulated by @adamreeve 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2310811588


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -527,28 +770,17 @@ async fn test_multi_threaded_encrypted_writing() {
 
     // Create a temporary file to write the encrypted data
     let temp_file = tempfile::tempfile().unwrap();
-    let mut writer = ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props).unwrap();
+    let mut writer = ArrowWriter::try_new(&temp_file, schema.clone(), props).unwrap();
 
     // LOW-LEVEL API: Use low level API to write into a file using multiple threads
 
     // Get column writers
+    #[allow(deprecated)]

Review Comment:
   That's a fair point! I introduced a new test that uses the new API (`test_multi_threaded_encrypted_writing_replace_deprecated_api`) to run besides the one using the old API one (`test_multi_threaded_encrypted_writing_deprecated`). They differ slightly (arrow_writers high-level API can't be used after the new low-level API borrows it's SerializedWriter) but I think we have the bases covered.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2358750140


##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -288,34 +288,17 @@ impl<W: AsyncFileWriter> AsyncArrowWriter<W> {
 
         Ok(())
     }
-
-    /// Create a new row group writer and return its column writers.
-    pub async fn get_column_writers(&mut self) -> Result<Vec<ArrowColumnWriter>> {

Review Comment:
   I double checked and this code is not yet released, so this is not a public API change. It was added in 
   
   - https://github.com/apache/arrow-rs/pull/8262 



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -906,6 +918,12 @@ impl ArrowRowGroupWriterFactory {
         let writers = get_column_writers(&self.schema, &self.props, &self.arrow_schema)?;
         Ok(ArrowRowGroupWriter::new(writers, &self.arrow_schema))
     }
+
+    /// Create column writers for a new row group.
+    pub fn create_column_writers(&self, row_group_index: usize) -> Result<Vec<ArrowColumnWriter>> {

Review Comment:
   I see now that this is the key API -- create column writers with the relevant encryption properties, if relevant



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -409,6 +409,7 @@ impl<W: Write + Send> ArrowWriter<W> {
     }
 
     /// Create a new row group writer and return its column writers.
+    #[deprecated(since = "56.2.0", note = "Use into_serialized_writer instead")]

Review Comment:
   Note to myself, these APIs were added in
   - https://github.com/apache/arrow-rs/pull/8029
   So they are relatively new



##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -361,7 +344,13 @@ mod tests {
         // Use classic API
         writer.write(&to_write_record).await.unwrap();
 
-        let mut writers = writer.get_column_writers().await.unwrap();
+        // Use low-level API to write an Arrow group
+        let arrow_writer = writer.sync_writer;

Review Comment:
   I think it would be good to use public APIs in the tests, rather than accessing the inner fields directly (which is not possible from other crates)



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].

Review Comment:
   I spent quite a while trying to figure out why we can't just use `get_column_writers` as in the [example](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel) to use ArrowRowGroupWriterFactory  and I (finally) realized the reason is the encryption configuration isn't passed to `get_column_writers`. `ArrowRowGroupWriterFactory` does have the encryption details and thus can make the correct ArrowColumnWriters.
   
   I think it is somewhat of a strange API to create an `ArrowWriter` only to immediately destructure it into a SerializedWriter / the underlying writer. It is also unfortunate we now have two different APIs for writing row groups in parallel, depending on encryption.
   
   I have an idea to make the APIs better as a follow on.



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].
+    /// This can be useful to provide more control over how files are written.
+    pub fn into_serialized_writer(

Review Comment:
   I spent quite some time trying to figure out why this API is needed -- specifically "why do we need an `ArrowWriter` at all, why not use SerializedFileWriter and `get_column_writers` directly, as shown in [this example](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel)
   
   After study I concluded the reason we need to expose `ArrowRowGroupWriterFactory` is that `ArrowRowGroupWriterFactory::create_column_writers` also has the appropriate encryption properties.
   
   It is unfortunate that we'll now have two different sets of APIs for creating column writers -- via `get_column_writers` AND `ArrowRowGroupWriterFactory`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3308257773

   Sorry for the late response @alamb and thanks for the added changes. I think this is good as is and can be included in the release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3293731445

   Hi @rok and @adamreeve  -- there appears to be a clippy error now (due to a new API that was added on main). Can you possibly take a look?
   
   Again, I apologize for the delay in review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2312640711


##########
parquet/tests/encryption/encryption_async.rs:
##########
@@ -527,28 +770,17 @@ async fn test_multi_threaded_encrypted_writing() {
 
     // Create a temporary file to write the encrypted data
     let temp_file = tempfile::tempfile().unwrap();
-    let mut writer = ArrowWriter::try_new(&temp_file, metadata.schema().clone(), props).unwrap();
+    let mut writer = ArrowWriter::try_new(&temp_file, schema.clone(), props).unwrap();
 
     // LOW-LEVEL API: Use low level API to write into a file using multiple threads
 
     // Get column writers
+    #[allow(deprecated)]

Review Comment:
   :+1: I think it's fine to not allow mixing the low-level and high-level APIs between different row groups like in the previous test, I can't imagine a use case for that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3241654853

   Thanks for the review @adamreeve ! I suppose it's time to ask @alamb to do a pass :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3310232231

   > I am sorry I haven't found time to study this PR in detail. @adamreeve it sounds like you have some good ideas and have devoted time here. What would you recommend?
   
   I've read through the latest comments and it sounds like you have a good understanding of the motivation for this change now, thanks for taking a close look Andrew.
   
   I agree it would be good to consolidate on using `ArrowRowGroupWriterFactory` for concurrent writing and update the examples. And also agree that creating an `ArrowWriter` just to convert it straight away to an `ArrowRowGroupWriterFactory` is a bit awkward. It does conveniently also create a `SerializedFileWriter` using the Arrow schema, but that's not too complicated to do explicitly, and the existing concurrent writing example already does this.
   
   If you'd rather not have the new `into_serialized_writer` method and require creating the `ArrowRowGroupWriterFactory` directly, maybe we should take some more time to implement your suggested approach and wait for the next release. But I think `into_serialized_writer` is convenient and unlikely to cause much extra maintenance burden.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#issuecomment-3293618115

   I merged up from main and plan to merge this PR in once the tests pass


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2362714998


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].

Review Comment:
   I filed a ticket with a suggestion for a unified API:
   - https://github.com/apache/arrow-rs/issues/8389



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -426,6 +428,15 @@ impl<W: Write + Send> ArrowWriter<W> {
         row_group_writer.close()?;
         Ok(())
     }
+
+    /// Converts this writer into a lower-level [`SerializedFileWriter`] and [`ArrowRowGroupWriterFactory`].
+    /// This can be useful to provide more control over how files are written.
+    pub fn into_serialized_writer(

Review Comment:
   I filed a ticket with a suggestion for a unified API:
   - https://github.com/apache/arrow-rs/issues/8389



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enable parallel writing across row groups when writing encrypted parquet [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8162:
URL: https://github.com/apache/arrow-rs/pull/8162#discussion_r2359237235


##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -349,51 +332,6 @@ mod tests {
         assert_eq!(to_write, read);
     }
 
-    #[tokio::test]

Review Comment:
   - I removed this test because it was added in https://github.com/apache/arrow-rs/pull/8262 along with `AsyncArrowWriter::get_column_writers` and `AsyncArrowWriter::append_row_group`
   
   Since we removed those APIs in this PR, we should also remove the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org