You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/09/16 10:49:25 UTC

[GitHub] [arrow-rs] alamb opened a new issue, #4823: API to copy an existing RowGroup, including metadata from one parquet file to another

alamb opened a new issue, #4823:
URL: https://github.com/apache/arrow-rs/issues/4823

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   In DataFusion, @devinjdangelo is using the [`append_column`](https://docs.rs/parquet/latest/parquet/file/writer/struct.SerializedRowGroupWriter.html#method.append_column) API to write parquet files in parallel (https://github.com/apache/arrow-datafusion/pull/7562) 
   
   However, when trying to copy the `RowGroupMetadata` to the API to copy any bloom filters / page offsets, or others is awkward
   
   
   **Describe the solution you'd like**
   
   I would like a way to to call the `append_column` api given a [`RowGroupMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.RowGroupMetaData.html) object from the existing file
   
   Ideally there would be an API that produced a [`ColumnCloseResult`](https://docs.rs/parquet/latest/parquet/column/writer/struct.ColumnCloseResult.html) from a `RowGroupMetaData` or some convenience API that took a reader + RowGroupMetadata from another file and did the necessary copy
   
   Perhaps something like
   
   ```rust
   impl SerializedRowGroupWriter {
   ...
     /// appends an entire RowGroup from the specified reader, including all
     /// metadata, to the in progress parquet file. 
     pub fn append_row_group(&mut self, rg: Box<dyn RowGroupReader>) -> Result<...> { 
      ...
     }
   }
   ```
    
   
   https://docs.rs/parquet/latest/parquet/file/writer/struct.SerializedRowGroupWriter.html#method.append_column
   
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features you've considered.
   -->
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] API to copy an existing RowGroup, including metadata from one parquet file to another [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4823:
URL: https://github.com/apache/arrow-rs/issues/4823#issuecomment-3437313020

   This is now handled -- see example in https://docs.rs/parquet/56.2.0/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb commented on issue #4823: API to copy an existing RowGroup, including metadata from one parquet file to another

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4823:
URL: https://github.com/apache/arrow-rs/issues/4823#issuecomment-1722473728

   >  Page and column indexes aren't stored at the row group level, so I'm not sure about this. We should definitely facilitate the use-case you describe, I'm not sure this is the way to do it
   
   It seems like the lowest level that makes sense when "copying parquet data from one file to another" is the `RowGroup`. Thus I think having an API in terms of the RowGroup makes sense (that is not to say there can't be more fine grained APIs at a lower level too). 🤔 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] API to copy an existing RowGroup, including metadata from one parquet file to another [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #4823: API to copy an existing RowGroup, including metadata from one parquet file to another
URL: https://github.com/apache/arrow-rs/issues/4823


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #4823: API to copy an existing RowGroup, including metadata from one parquet file to another

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4823:
URL: https://github.com/apache/arrow-rs/issues/4823#issuecomment-1722217438

   Page and column indexes aren't stored at the row group level, so I'm not sure about this. We should definitely facilitate the use-case you describe, I'm not sure this is the way to do it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #4823: API to copy an existing RowGroup, including metadata from one parquet file to another

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4823:
URL: https://github.com/apache/arrow-rs/issues/4823#issuecomment-1722481975

   I think my comment was ambiguous, `RowGroupMetadata` doesn't contain page indices, I therefore don't think it can be the basis of this API. I have no objections to the notion of logically appending row groups, purely your suggested mechanism


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org