You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "stuartcarnie (via GitHub)" <gi...@apache.org> on 2023/02/28 19:20:16 UTC
[GitHub] [arrow-rs] stuartcarnie opened a new issue, #3779: Schema-level metadata is not encoded in Flight responses
stuartcarnie opened a new issue, #3779:
URL: https://github.com/apache/arrow-rs/issues/3779
**Describe the bug**
When preparing schema for encoding into a Flight response, the schema-level metadata from the source schema is dropped:
https://github.com/apache/arrow-rs/blob/e7eb304dac442a943c434f8ea248de909f82aa88/arrow-flight/src/encode.rs#L326
**To Reproduce**
```patch
Index: arrow-flight/src/encode.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/arrow-flight/src/encode.rs b/arrow-flight/src/encode.rs
--- a/arrow-flight/src/encode.rs (revision e7eb304dac442a943c434f8ea248de909f82aa88)
+++ b/arrow-flight/src/encode.rs (date 1677611826249)
@@ -453,6 +453,7 @@
#[cfg(test)]
mod tests {
+ use std::collections::HashMap;
use arrow::{
array::{UInt32Array, UInt8Array},
compute::concat_batches,
@@ -502,6 +503,16 @@
);
}
+ #[test]
+ fn test_schema_metadata_encoded() {
+ let schema = Schema::new(vec![
+ Field::new("data", DataType::Int32, false),
+ ]).with_metadata(HashMap::from([("some_key".to_owned(), "some_value".to_owned())]));
+
+ let got = prepare_schema_for_flight(&schema);
+ assert!(got.metadata().contains_key("some_key"));
+ }
+
#[test]
fn test_encode_no_column_batch() {
let batch = RecordBatch::try_new_with_options(
```
**Expected behavior**
Schema-level metadata should be included permitting test to pass.
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3779: Schema-level metadata is not encoded in Flight responses
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #3779:
URL: https://github.com/apache/arrow-rs/issues/3779#issuecomment-1463939956
`label_issue.py` automatically added labels {'arrow'} from #3811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3779: Schema-level metadata is not encoded in Flight responses
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #3779:
URL: https://github.com/apache/arrow-rs/issues/3779#issuecomment-1463940005
`label_issue.py` automatically added labels {'arrow-flight'} from #3811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] stuartcarnie commented on issue #3779: Schema-level metadata is not encoded in Flight responses
Posted by "stuartcarnie (via GitHub)" <gi...@apache.org>.
stuartcarnie commented on issue #3779:
URL: https://github.com/apache/arrow-rs/issues/3779#issuecomment-1451046234
@alamb I can confirm that with this patch:
```patch
Index: arrow-flight/src/encode.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/arrow-flight/src/encode.rs b/arrow-flight/src/encode.rs
--- a/arrow-flight/src/encode.rs (revision bbc1469077e13ba2e5a61f130917ad7eccfcb569)
+++ b/arrow-flight/src/encode.rs (date 1677712010122)
@@ -323,7 +323,7 @@
})
.collect();
- Schema::new(fields)
+ Schema::new(fields).with_metadata(schema.metadata().clone())
}
/// Split [`RecordBatch`] so it hopefully fits into a gRPC response.
@@ -453,6 +453,7 @@
#[cfg(test)]
mod tests {
+ use std::collections::HashMap;
use arrow::{
array::{UInt32Array, UInt8Array},
compute::concat_batches,
@@ -502,6 +503,16 @@
);
}
+ #[test]
+ fn test_schema_metadata_encoded() {
+ let schema = Schema::new(vec![
+ Field::new("data", DataType::Int32, false),
+ ]).with_metadata(HashMap::from([("some_key".to_owned(), "some_value".to_owned())]));
+
+ let got = prepare_schema_for_flight(&schema);
+ assert!(got.metadata().contains_key("some_key"));
+ }
+
#[test]
fn test_encode_no_column_batch() {
let batch = RecordBatch::try_new_with_options(
```
and minor fixes in IOx, the schema metadata propagates across RPC requests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #3779: Schema-level metadata is not encoded in Flight responses
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #3779: Schema-level metadata is not encoded in Flight responses
URL: https://github.com/apache/arrow-rs/issues/3779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org