You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jecsand838 (via GitHub)" <gi...@apache.org> on 2025/04/28 22:18:27 UTC

[PR] Add Map support to arrow-avro [arrow-rs]

jecsand838 opened a new pull request, #7451:
URL: https://github.com/apache/arrow-rs/pull/7451

   # Which issue does this PR close?
   
   Part of https://github.com/apache/arrow-rs/issues/4886
   
   Related to https://github.com/apache/arrow-rs/pull/6965
   
   # Rationale for this change
    
   Avro supports maps as a core data type, but previously arrow-avro lacked any decoding logic to handle them. As a result, any Avro file containing map fields would fail to parse correctly within the Arrow ecosystem. This PR addresses this gap by:
   
   1. Adding explicit Map decoding: It introduces a new` Codec::Map` variant and corresponding `Decoder::Map` logic that reads map blocks in Avro format and constructs an Arrow MapArray. This preserves both the keys and values in a schema-compatible manner, enabling accurate ingestion of map-encoded data.
   
   Overall, these changes expand Arrow’s Avro reader capabilities, allowing users to work with map-encoded data in a standardized Arrow format.
   
   # What changes are included in this PR?
   
   This PR adds full Map support in the arrow-avro crate. Key changes include:
   
   **1. arrow-avro/src/codec.rs:**
   
   * Added a `Codec::Map(Arc<AvroDataType>)` variant
   * Logic to construct an Arrow MapArray, including a struct of { key, value } with typed value decoding.
   * Added a new function to `AvroDataType`.
   
   **2. arrow-avro/src/reader/record.rs:**
   
   * Introduced the Map decoding path which leverages blockwise reads of Avro map data.
   * Created helpers read_map_blocks and `read_blockwise_items` to handle block sequences of key value pairs until a terminating block count of zero.
   * Implemented decoder unit tests for Map types.
   
   # Are there any user-facing changes?
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2905445096

   > I've not reviewed the code, but as this is not actually public (none of the avro crate is) it doesn't need to wait for a breaking release
   
   In that case, let's get it merged!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2069291552


##########
arrow-avro/src/codec.rs:
##########
@@ -155,6 +168,22 @@ impl Codec {
                 DataType::List(Arc::new(f.field_with_name(Field::LIST_FIELD_DEFAULT_NAME)))
             }
             Self::Struct(f) => DataType::Struct(f.iter().map(|x| x.field()).collect()),
+            Self::Map(value_type) => {
+                let val_dt = value_type.codec.data_type();
+                let val_field = Field::new("value", val_dt, true)

Review Comment:
   @klion26  That's a really good call out. I think it does make sense to add that in here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2843204435

   @klion26 I pushed changes just now that:
   
   * Fixed the linter errors
   * Made `read_blockwise_items` more readable
   * Changed `let val_field = Field::new("value", val_dt, true)` to  `let val_field = Field::new("value", val_dt, value_type.nullability.is_some())`
   
   Let me know what you think!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2905446520

   Thanks @jecsand838  -- it is exciting to see progress here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2069363906


##########
arrow-avro/src/reader/record.rs:
##########
@@ -267,10 +305,83 @@ impl Decoder {
                     .collect::<Result<Vec<_>, _>>()?;
                 Arc::new(StructArray::new(fields.clone(), arrays, nulls))
             }
+            Self::Map(map_field, k_off, m_off, kdata, valdec) => {
+                let moff = flush_offsets(m_off);
+                let koff = flush_offsets(k_off);
+                let kd = flush_values(kdata).into();
+                let val_arr = valdec.flush(None)?;
+                let key_arr = StringArray::new(koff, kd, None);
+                if key_arr.len() != val_arr.len() {
+                    return Err(ArrowError::InvalidArgumentError(format!(
+                        "Map keys length ({}) != map values length ({})",
+                        key_arr.len(),
+                        val_arr.len()
+                    )));
+                }
+                let final_len = moff.len() - 1;
+                if let Some(n) = &nulls {
+                    if n.len() != final_len {
+                        return Err(ArrowError::InvalidArgumentError(format!(
+                            "Map array null buffer length {} != final map length {final_len}",
+                            n.len()
+                        )));
+                    }
+                }
+                let entries_struct = StructArray::new(
+                    Fields::from(vec![
+                        Arc::new(ArrowField::new("key", DataType::Utf8, false)),
+                        Arc::new(ArrowField::new("value", val_arr.data_type().clone(), true)),
+                    ]),
+                    vec![Arc::new(key_arr), val_arr],
+                    None,
+                );
+                let map_arr = MapArray::new(map_field.clone(), moff, entries_struct, nulls, false);
+                Arc::new(map_arr)
+            }
         })
     }
 }
 
+
+fn read_map_blocks(
+    buf: &mut AvroCursor,
+    decode_entry: impl FnMut(&mut AvroCursor) -> Result<(), ArrowError>,
+) -> Result<usize, ArrowError> {
+    read_blockwise_items(buf, true, decode_entry)
+}
+
+fn read_blockwise_items(
+    buf: &mut AvroCursor,
+    read_size_after_negative: bool,
+    mut decode_fn: impl FnMut(&mut AvroCursor) -> Result<(), ArrowError>,
+) -> Result<usize, ArrowError> {
+    let mut total = 0usize;
+    loop {
+        let blk = buf.get_long()?;

Review Comment:
   @klion26 
   
   `blk` is the block count and there are cases where `blk` will be negative. 
   
   A negative `blk` is expected by the Avro spec for block-encoded arrays & maps, indicating that the count is `-blk` items. Avro decoders usually handle this by reading the size marker and then proceeding to decode `|blk|` entries in that block. After finishing the block, decoding continues with the next block count, until a 0 count terminates the sequence.
   
   Here's the text from the [Avro Specification](https://avro.apache.org/docs/1.11.1/specification/#maps) regarding maps with negative block counts:
   
   > If a block’s count is negative, its absolute value is used, and the count is followed immediately by a long block size indicating the number of bytes in the block. This block size permits fast skipping through data, e.g., when projecting a record to a subset of its fields.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2065764967


##########
arrow-avro/src/codec.rs:
##########
@@ -284,9 +313,14 @@ fn make_data_type<'a>(
             ComplexType::Enum(e) => Err(ArrowError::NotYetImplemented(format!(
                 "Enum of {e:?} not currently supported"
             ))),
-            ComplexType::Map(m) => Err(ArrowError::NotYetImplemented(format!(
-                "Map of {m:?} not currently supported"
-            ))),
+            ComplexType::Map(m) => {
+                let val = make_data_type(&m.values, namespace, resolver)?;
+                Ok(AvroDataType {
+                    nullability: None,

Review Comment:
   Do we need to set the `nullability` to `val.nullability`?



##########
arrow-avro/src/codec.rs:
##########
@@ -155,6 +168,22 @@ impl Codec {
                 DataType::List(Arc::new(f.field_with_name(Field::LIST_FIELD_DEFAULT_NAME)))
             }
             Self::Struct(f) => DataType::Struct(f.iter().map(|x| x.field()).collect()),
+            Self::Map(value_type) => {
+                let val_dt = value_type.codec.data_type();
+                let val_field = Field::new("value", val_dt, true)

Review Comment:
   Does the `nullability` set to `value_type.nullability.is_some()`



##########
arrow-avro/src/reader/record.rs:
##########
@@ -290,3 +401,84 @@ fn flush_primitive<T: ArrowPrimitiveType>(
 }
 
 const DEFAULT_CAPACITY: usize = 1024;
+
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use arrow_array::{
+        cast::AsArray, Array, Decimal128Array, DictionaryArray, FixedSizeBinaryArray,
+        IntervalMonthDayNanoArray, ListArray, MapArray, StringArray, StructArray,
+    };
+
+    fn encode_avro_long(value: i64) -> Vec<u8> {
+        let mut buf = Vec::new();
+        let mut v = (value << 1) ^ (value >> 63);
+        while v & !0x7F != 0 {
+            buf.push(((v & 0x7F) | 0x80) as u8);
+            v >>= 7;
+        }
+        buf.push(v as u8);
+        buf
+    }
+
+    fn encode_avro_bytes(bytes: &[u8]) -> Vec<u8> {
+        let mut buf = encode_avro_long(bytes.len() as i64);
+        buf.extend_from_slice(bytes);
+        buf
+    }
+
+    fn avro_from_codec(codec: Codec) -> AvroDataType {
+        AvroDataType::new(codec, None, Default::default())

Review Comment:
   Not sure if we need to test the `nullability` to `Some(x)`



##########
arrow-avro/src/reader/record.rs:
##########
@@ -267,10 +305,83 @@ impl Decoder {
                     .collect::<Result<Vec<_>, _>>()?;
                 Arc::new(StructArray::new(fields.clone(), arrays, nulls))
             }
+            Self::Map(map_field, k_off, m_off, kdata, valdec) => {
+                let moff = flush_offsets(m_off);
+                let koff = flush_offsets(k_off);
+                let kd = flush_values(kdata).into();
+                let val_arr = valdec.flush(None)?;
+                let key_arr = StringArray::new(koff, kd, None);
+                if key_arr.len() != val_arr.len() {
+                    return Err(ArrowError::InvalidArgumentError(format!(
+                        "Map keys length ({}) != map values length ({})",
+                        key_arr.len(),
+                        val_arr.len()
+                    )));
+                }
+                let final_len = moff.len() - 1;
+                if let Some(n) = &nulls {
+                    if n.len() != final_len {
+                        return Err(ArrowError::InvalidArgumentError(format!(
+                            "Map array null buffer length {} != final map length {final_len}",
+                            n.len()
+                        )));
+                    }
+                }
+                let entries_struct = StructArray::new(
+                    Fields::from(vec![
+                        Arc::new(ArrowField::new("key", DataType::Utf8, false)),
+                        Arc::new(ArrowField::new("value", val_arr.data_type().clone(), true)),
+                    ]),
+                    vec![Arc::new(key_arr), val_arr],
+                    None,
+                );
+                let map_arr = MapArray::new(map_field.clone(), moff, entries_struct, nulls, false);
+                Arc::new(map_arr)
+            }
         })
     }
 }
 
+
+fn read_map_blocks(
+    buf: &mut AvroCursor,
+    decode_entry: impl FnMut(&mut AvroCursor) -> Result<(), ArrowError>,
+) -> Result<usize, ArrowError> {
+    read_blockwise_items(buf, true, decode_entry)
+}
+
+fn read_blockwise_items(
+    buf: &mut AvroCursor,
+    read_size_after_negative: bool,
+    mut decode_fn: impl FnMut(&mut AvroCursor) -> Result<(), ArrowError>,
+) -> Result<usize, ArrowError> {
+    let mut total = 0usize;
+    loop {
+        let blk = buf.get_long()?;

Review Comment:
   It seems `blk` is the number of items needed to be read. Is there any case where `blk` will be negative?



##########
arrow-avro/src/codec.rs:
##########
@@ -155,6 +168,22 @@ impl Codec {
                 DataType::List(Arc::new(f.field_with_name(Field::LIST_FIELD_DEFAULT_NAME)))
             }
             Self::Struct(f) => DataType::Struct(f.iter().map(|x| x.field()).collect()),
+            Self::Map(value_type) => {
+                let val_dt = value_type.codec.data_type();
+                let val_field = Field::new("value", val_dt, true)

Review Comment:
   Read the [spec](https://avro.apache.org/docs/1.11.1/specification/#schema-declaration), did not find that if the value for `Map` can be null or not, but asked Deepseek, it says that the value can be null if the schema supports it(use `Unions` such as `["null", "string"]`), else it can't be null.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2080226794


##########
arrow-avro/src/codec.rs:
##########
@@ -162,6 +175,8 @@ pub enum Codec {
     List(Arc<AvroDataType>),
     /// Represents Avro record type, maps to Arrow's Struct data type
     Struct(Arc<[AvroField]>),
+    /// Represents Avro map type, maps to Arrow's Map data type

Review Comment:
   Since this is a `pub` enum and it is not marked as `non exhaustive` I think this is technically a breaking API change and will need to wait for the next major release:
   - #7395 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2069285439


##########
arrow-avro/src/codec.rs:
##########
@@ -155,6 +168,22 @@ impl Codec {
                 DataType::List(Arc::new(f.field_with_name(Field::LIST_FIELD_DEFAULT_NAME)))
             }
             Self::Struct(f) => DataType::Struct(f.iter().map(|x| x.field()).collect()),
+            Self::Map(value_type) => {
+                let val_dt = value_type.codec.data_type();
+                let val_field = Field::new("value", val_dt, true)

Review Comment:
   @klion26 
   
   I was attempting to follow the same pattern used in the other types such as `ComplexType::Record(r)`, `ComplexType::Array(a)`, and `ComplexType::Fixed(f)`.
   
   It seemed from reading the code that nullability would only be set in the `Schema::Union(f)` branch of `make_data_type`.



##########
arrow-avro/src/codec.rs:
##########
@@ -284,9 +313,14 @@ fn make_data_type<'a>(
             ComplexType::Enum(e) => Err(ArrowError::NotYetImplemented(format!(
                 "Enum of {e:?} not currently supported"
             ))),
-            ComplexType::Map(m) => Err(ArrowError::NotYetImplemented(format!(
-                "Map of {m:?} not currently supported"
-            ))),
+            ComplexType::Map(m) => {
+                let val = make_data_type(&m.values, namespace, resolver)?;
+                Ok(AvroDataType {
+                    nullability: None,

Review Comment:
   @klion26 
   
   I was attempting to follow the same pattern used in the other types such as `ComplexType::Record(r)`, `ComplexType::Array(a)`, and `ComplexType::Fixed(f)`.
   
   It seemed from reading the code that nullability would only be set in the `Schema::Union(f)` branch of `make_data_type`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2069301291


##########
arrow-avro/src/reader/record.rs:
##########
@@ -290,3 +401,84 @@ fn flush_primitive<T: ArrowPrimitiveType>(
 }
 
 const DEFAULT_CAPACITY: usize = 1024;
+
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use arrow_array::{
+        cast::AsArray, Array, Decimal128Array, DictionaryArray, FixedSizeBinaryArray,
+        IntervalMonthDayNanoArray, ListArray, MapArray, StringArray, StructArray,
+    };
+
+    fn encode_avro_long(value: i64) -> Vec<u8> {
+        let mut buf = Vec::new();
+        let mut v = (value << 1) ^ (value >> 63);
+        while v & !0x7F != 0 {
+            buf.push(((v & 0x7F) | 0x80) as u8);
+            v >>= 7;
+        }
+        buf.push(v as u8);
+        buf
+    }
+
+    fn encode_avro_bytes(bytes: &[u8]) -> Vec<u8> {
+        let mut buf = encode_avro_long(bytes.len() as i64);
+        buf.extend_from_slice(bytes);
+        buf
+    }
+
+    fn avro_from_codec(codec: Codec) -> AvroDataType {
+        AvroDataType::new(codec, None, Default::default())

Review Comment:
   @klion26 I was thinking it would be best to handle nullability in a dedicated PR after the basic support for each type is merged in. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2905598149

   This CI broke main due to a logical conflict, here is a fix:
    - https://github.com/apache/arrow-rs/pull/7542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on code in PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#discussion_r2069285439


##########
arrow-avro/src/codec.rs:
##########
@@ -155,6 +168,22 @@ impl Codec {
                 DataType::List(Arc::new(f.field_with_name(Field::LIST_FIELD_DEFAULT_NAME)))
             }
             Self::Struct(f) => DataType::Struct(f.iter().map(|x| x.field()).collect()),
+            Self::Map(value_type) => {
+                let val_dt = value_type.codec.data_type();
+                let val_field = Field::new("value", val_dt, true)

Review Comment:
   @klion26 
   
   I was attempting to follow the same pattern used in the other types such as `ComplexType::Record(r)`, `ComplexType::Array(a)`, and `ComplexType::Fixed(f)`.
   
   It seemed from reading the code that nullability would only be set in the `Schema::Union(f)` branch of `make_data_type`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2904102050

   I've not reviewed the code, but as this is not actually public (none of the avro crate is) it doesn't need to wait for a breaking release


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2913480899

   @alamb @tustvold Absolutely! We'll go ahead and get some more PRs up this week then! Thank you all much for getting this in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Map support to arrow-avro [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7451:
URL: https://github.com/apache/arrow-rs/pull/7451#issuecomment-2913505097

   > @alamb @tustvold Absolutely! We'll go ahead and get some more PRs up this week then! Thank you all so much for getting this in.
   
   Thanks! And now that I know we don't have to worry about backwards compatibility I think we can iterate more quickly here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org