You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "evenyag (via GitHub)" <gi...@apache.org> on 2024/01/11 14:36:37 UTC

[I] parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter` [arrow-rs]

evenyag opened a new issue, #5296:
URL: https://github.com/apache/arrow-rs/issues/5296

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   <!--
   A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 
   (This section helps Arrow developers understand the context and *why* for this feature, in addition to  the *what*)
   -->
   
   The `ArrowWriter` always adds the metadata `ARROW_SCHEMA_META_KEY` to its generated parquet file.
   https://github.com/apache/arrow-rs/blob/72d8a783176219f0864022daba70e84ceab7e221/parquet/src/arrow/arrow_writer/mod.rs#L118-L126
   
   Sometimes I'd like to write a parquet file without additional metadata or I have another metadata to describe the schema. It'd be helpful to provide a way to disable the embed metadata.
   
   **Describe the solution you'd like**
   <!--
   A clear and concise description of what you want to happen.
   -->
   
   I found that the CPP implementation disables the metadata by default and provides a `store_schema` option to enable it.
   - https://github.com/apache/arrow/blob/main/docs/source/cpp/parquet.rst#id61
   - https://arrow.apache.org/docs/dev/cpp/api/formats.html#_CPPv4N7parquet21ArrowWriterProperties7Builder12store_schemaEv
   
   For backward compatibility, we could add an `ArrowWriterOptions` and enable arrow metadata by default. The `ArrowWriter` has a new API to construct itself with the options.
   ```rust
   struct ArrowWriterOptions {}
   
   impl ArrowWriterOptions {
       pub fn new() -> Self {}
   
       pub fn with_skip_arrow_metadata(self, skip_arrow_metadata: bool) -> Self {}
   }
   
   impl<W> ArrowWriter<W> {
       pub fn try_new_with_options(
           writer: W,
           arrow_schema: SchemaRef,
           props: Option<WriterProperties>,
           options: ArrowReaderOptions,
       ) -> Result<Self> {}
   }
   ```
   
   If `skip_arrow_metadata` is true, the writer won't store the arrow schema meta.
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features you've considered.
   -->
   
   We could also use the same option `store_schema` as arrow-cpp and set it as true by default.
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5296: parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter`
URL: https://github.com/apache/arrow-rs/issues/5296


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter` [arrow-rs]

Posted by "evenyag (via GitHub)" <gi...@apache.org>.
evenyag commented on issue #5296:
URL: https://github.com/apache/arrow-rs/issues/5296#issuecomment-1888355793

   I would like to have a try


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5296:
URL: https://github.com/apache/arrow-rs/issues/5296#issuecomment-1887384180

   Adding an option to skip embedding the arrow metadata makes sense to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet: Allow disabling embed `ARROW_SCHEMA_META_KEY` added by the `ArrowWriter` [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5296:
URL: https://github.com/apache/arrow-rs/issues/5296#issuecomment-1972588581

   `label_issue.py` automatically added labels {'parquet'} from #5299


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org