[GitHub] [arrow-rs] alamb commented on a diff in pull request #1566: Read/write nested dictionary in ipc stream reader/writer

[GitBox <gi...@apache.org>]

alamb commented on code in PR #1566:
URL: https://github.com/apache/arrow-rs/pull/1566#discussion_r851256154


##########
arrow/src/datatypes/field.rs:
##########
@@ -116,14 +116,25 @@ impl Field {
     /// Returns a (flattened) vector containing all fields contained within this field (including it self)
     pub(crate) fn fields(&self) -> Vec<&Field> {
         let mut collected_fields = vec![self];
-        match &self.data_type {
+        collected_fields.append(&mut self._fields(&self.data_type));
+
+        collected_fields
+    }
+
+    fn _fields<'a>(&'a self, dt: &'a DataType) -> Vec<&Field> {
+        let mut collected_fields = vec![];
+
+        match dt {
             DataType::Struct(fields) | DataType::Union(fields, _) => {
                 collected_fields.extend(fields.iter().flat_map(|f| f.fields()))
             }
             DataType::List(field)
             | DataType::LargeList(field)
             | DataType::FixedSizeList(field, _)
             | DataType::Map(field, _) => collected_fields.push(field),
+            DataType::Dictionary(_, value_field) => {
+                collected_fields.append(&mut self._fields(value_field.as_ref()))
+            }

Review Comment:
   I wonder if it is worth adding an error for types that aren't explicitly supported (I realize this will likely need a bunch of plumbing to return `Result`)
   
   like
   ```rust
   _ => return Err("Type not supported")
   ```



##########
arrow/src/datatypes/field.rs:
##########
@@ -116,14 +116,25 @@ impl Field {
     /// Returns a (flattened) vector containing all fields contained within this field (including it self)
     pub(crate) fn fields(&self) -> Vec<&Field> {
         let mut collected_fields = vec![self];
-        match &self.data_type {
+        collected_fields.append(&mut self._fields(&self.data_type));
+
+        collected_fields
+    }
+
+    fn _fields<'a>(&'a self, dt: &'a DataType) -> Vec<&Field> {
+        let mut collected_fields = vec![];
+
+        match dt {
             DataType::Struct(fields) | DataType::Union(fields, _) => {
                 collected_fields.extend(fields.iter().flat_map(|f| f.fields()))
             }
             DataType::List(field)
             | DataType::LargeList(field)
             | DataType::FixedSizeList(field, _)
             | DataType::Map(field, _) => collected_fields.push(field),
+            DataType::Dictionary(_, value_field) => {
+                collected_fields.append(&mut self._fields(value_field.as_ref()))

Review Comment:
   the key improvement of the PR is that this now recurses into children, right?



##########
arrow/src/ipc/writer.rs:
##########
@@ -175,13 +186,37 @@ impl IpcDataGenerator {
                     )?;
                 }
             }
+            _ => (),

Review Comment:
   maybe this should return an error for unsupported nested types



##########
arrow/src/ipc/reader.rs:
##########
@@ -1439,4 +1440,42 @@ mod tests {
         let output_batch = roundtrip_ipc_stream(&input_batch);
         assert_eq!(input_batch, output_batch);
     }
+
+    #[test]
+    fn test_roundtrip_stream_nested_dict_dict() {

Review Comment:
   I tried running this test without the code changes in this PR and it failed thusly. 👍 
   
   ```
   ---- ipc::reader::tests::test_roundtrip_stream_nested_dict_dict stdout ----
   thread 'ipc::reader::tests::test_roundtrip_stream_nested_dict_dict' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("dictionary id not found in schema")', arrow/src/ipc/reader.rs:1331:32
   ```
   when I removed the field code
   
    
   and  like this when I removed the writer support
   ```
   ---- ipc::reader::tests::test_roundtrip_stream_nested_dict_dict stdout ----
   thread 'ipc::reader::tests::test_roundtrip_stream_nested_dict_dict' panicked at 'called `Option::unwrap()` on a `None` value', arrow/src/ipc/reader.rs:176:64
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org