[I] Improve ability for users to create their own writers using the low level parquet APIs. [arrow-rs]

["wiedld (via GitHub)" <gi...@apache.org>]

wiedld opened a new issue, #6177:
URL: https://github.com/apache/arrow-rs/issues/6177

   **Is your feature request related to a ~problem or~ challenge? Please describe what you are trying to do.**
   
   We have been using at least two parquet writers that both utilize the low-level APIs provided by the parquet crate (e.g. [SerializedFileWriter](https://docs.rs/parquet/52.2.0/parquet/file/writer/struct.SerializedFileWriter.html)). One of the writers (ArrowWriter) is provided as part of the parquet crate, whereas the other parallel writer (datafusion's ParquetSink) is not. However, in both cases we later attempt to read these files using the parquet crate's readers.
   
   The challenge is that we keep encountering unexpected differences in the parquet written by these two writers. The most recent example is that the [arrow schema is missing when using datafusion's parallel writer](https://github.com/apache/datafusion/issues/11770), whereas it is [included in the ArrowWriter](ttps://github.com/apache/arrow-rs/blob/2905ce6796cad396241fc50164970dbf1237440a/parquet/src/arrow/arrow_writer/mod.rs#L188-L190) on parquet write.
   
   
   **Describe the solution you'd like**
   
   Can we update the lower level APIs (in the parquet crate) to make it easier for users to create their own parquet writers -- without encountering surprise differences from the behavior of parquet's ArrowWriter? Provide better documentation? Provide guidance for testing when creating your own parquet writer?
   
   
   **Describe alternatives you've considered**
   
   Alternatively, we could consider this problem as the responsibility for the users that create their own writers. We already plan to file a datafusion ticket proposing that we need integration tests to ensure byte equivalency in the output parquet (vs parquet written by ArrowWriter).
   
   **Additional context**
   
   This is not the first time that we have discovered differences in the output parquet from the ArrowWriter vs datafusion's ParquetSink. However, we are unclear on the best way to divide responsibilities (for more testing) vs API design.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve ability for users to create their own writers using the low level parquet APIs. [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #6177:
URL: https://github.com/apache/arrow-rs/issues/6177#issuecomment-2263857306

   In particular DataFusion uses the `ArrowColumnWriter`: https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html
   
   The example on that structure 
   
   ```rust
   
   // Compute the parquet schema
   let parquet_schema = arrow_to_parquet_schema(schema.as_ref()).unwrap();
   let props = Arc::new(WriterProperties::default());
   ...
   
   Which doesn't encode the arrow schema in the metadata of `props`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org