You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/02/10 13:25:41 UTC
[GitHub] [arrow-datafusion] jiacai2050 opened a new issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
jiacai2050 opened a new issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806
**Describe the bug**
```rust
// DataFusion
pub struct DFSchema {
/// Fields
fields: Vec<DFField>,
}
// Arrow
pub struct Schema {
pub(crate) fields: Vec<Field>,
/// A map of key-value pairs containing additional meta data.
#[serde(skip_serializing_if = "HashMap::is_empty")]
#[serde(default)]
pub(crate) metadata: HashMap<String, String>,
}
```
There is no place to keep metadata
**Expected behavior**
metadata should be kept during convert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806#issuecomment-1036082879
I am not sure we use the metadata attached to schema anywhere in datafusion. However, I think it is common for users of arrow or datafusion to put application specific information into that metadata (for example, IOx stores information about sortedness and other IOx specific things in the schema)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806#issuecomment-1035517966
👍 this would be nice to fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
Posted by GitBox <gi...@apache.org>.
alamb closed issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806#issuecomment-1035625546
I am thinking if we add it, where to use it in DF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiacai2050 commented on issue #1806: Metadata of Arrow Schema is lost when converted to DFSchema
Posted by GitBox <gi...@apache.org>.
jiacai2050 commented on issue #1806:
URL: https://github.com/apache/arrow-datafusion/issues/1806#issuecomment-1036197104
@alamb Yeap, my project also use metadata to keep application specific info.
I will try submit a PR fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org