You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "emilk (via GitHub)" <gi...@apache.org> on 2026/06/04 13:26:18 UTC

[I] ref-counted `Metadata` type [arrow-rs]

emilk opened a new issue, #10069:
URL: https://github.com/apache/arrow-rs/issues/10069

   ### Background
   Both `RecordBatch/Schema` and `Field` can have `metadata`. In both cases they are encoded as `HashMap<String, String>`
   
   One downside with this is that cloning the metadata is slow (requires a deep clone and a lot of allocations). This is in contrast with basically everything else in `arrow-rs`, which uses an `Arc` for fast cloning.
   
   ### Proposal outline
   
   ```rs
   #[derive(Clone, Default, …)]
   pub struct Metadata(
       // Use `Option` to avoid allocation in case of empty metadata
       Option<Arc<BTreeMap<String, String>>>
   )
   
   impl Metdata {
       pub fn get(&self, key: &str) -> Option<&String> { … }
   
       /// Does deep clone if (and only if) this `Metadata` is shared
       pub fn insert(&mut self, key: impl Into<String>, value: impl Into<String>) {
           Arc::make_mut(self.0.get_or_insert_default()).insert(key.into(), value.into());
       }
   
       …
   }
   
   impl Index<…> for Metadata …
   
   impl From<HashMap<String, String>> for Metadata { … }
   impl From<BTreeMap<String, String>> for Metadata { … }
   impl Into<HashMap<String, String>> for Metadata { … }
   impl Into<BTreeMap<String, String>> for Metadata { … }
   
   impl IntoIterator, FromIterator, …
   ```
   
   ### PRO/CON vs status quo (`HashMap<String, String>`)
   * PRO: Fast cloning of the whole `Metadata`
   * PRO: Deterministic iteration order (thanks to `BTreeMap`) - good for IPC/FFI encoding, test stability, hashing, …
   * NEUTRAL: Can still add/remove `Metadata` fields without extra cost
   * CON: New type; more complexity
   
   
   ### Alternatives
   Instead of storing `String`, we could store `Arc<str>`. That would make it efficient to share the same keys across many metadata tables.
   
   The downside is added complexity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ref-counted `Metadata` type [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10069:
URL: https://github.com/apache/arrow-rs/issues/10069#issuecomment-4624367145

   I think this is a good idea. thank you @emilk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ref-counted `Metadata` type [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10069:
URL: https://github.com/apache/arrow-rs/issues/10069#issuecomment-4624363309

   Makes sense to me
   
   Also related is the following (where we came to the same "let's use `Arc<HashMap<..>>`") solution
   - https://github.com/apache/arrow-rs/pull/9445


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ref-counted `Metadata` type [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl closed issue #10069: ref-counted `Metadata` type
URL: https://github.com/apache/arrow-rs/issues/10069


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] ref-counted `Metadata` type [arrow-rs]

Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on issue #10069:
URL: https://github.com/apache/arrow-rs/issues/10069#issuecomment-4624743048

   Then I'll get started on it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org