You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Kriskras99 (via GitHub)" <gi...@apache.org> on 2026/03/20 10:49:04 UTC
[PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Kriskras99 opened a new pull request, #512:
URL: https://github.com/apache/avro-rs/pull/512
I've done my best to make this PR as small as possible, but the serializer and deserializer implementations need to match. So I can't make it smaller than this. Every commit builds, but only the last commit is guaranteed to pass tests.
This is a breaking change, as this makes the serializer as strict as possible (checking names). An array is now also serialized as a tuple (like Serde does) unless `apache_avro::serde::array` is used.
Other stuff that happens in this PR:
- Document the mapping between the Serde and Avro data models
- `human_writable` is now configurable per reader and writer instance
- `Array` and `Map` serialisation will not write the block size in bytes unless configured to do so
- This means they don't need to buffer the data, causing performance improvements
- Fix the `AvroSchemaComponent` implementation of `std::time::Duration`
- The original implementation didn't work as the `Serialize` implementation for this type is a struct
- Fix the `AvroSchemaComponent` for `bigdecimal::BigDecimal`
- The `Serialize` implementation for `BigDecimal` is a string, which had a workaround in the serializer to make it serialize to `Schema::BigDecimal`. Now the default is `Schema::String` which can be overwritten using `apache_avro::serde::bigdecimal`
This PR does not include support in `avro_schema_derive`, that will come in a separate PR. I also have a deserializer implementation in the works that can resolve schemas, which should allow use the completely deprecate `from_value` and `to_value`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4162213147
Rebased onto main, Clippy lints enabled and fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969305777
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1688 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod array;
+mod enums;
+mod identifier;
+mod map;
+mod record;
+mod tuple;
+
+use array::ArrayDeserializer;
+use enums::PlainEnumDeserializer;
+use map::MapDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let variant = schema.get_variant(index as usize)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
Review Comment:
Unfortunately yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4152888129
Understandable, just wanted to make sure there's nothing on my side to do.
Take your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-5553645791
Hey @Kriskras99 thanks for adding this! This is exactly what I'm looking for 😄
It works really well for the upstream `iceberg-rust` manifest reader, since we already know the target Rust types and don’t need to go through `Value` first. Nice to be able to keep using `apache-avro` here without the extra conversion step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3015908596
##########
avro/src/reader/single_object.rs:
##########
@@ -315,11 +354,11 @@ mod tests {
fn avro_rs_164_generic_reader_alternate_header() -> TestResult {
let schema_uuid = Uuid::parse_str("b2f1cf00-0434-013e-439a-125eb8485a5f")?;
let header_builder = GlueSchemaUuidHeader::from_uuid(schema_uuid);
- let generic_reader = GenericSingleObjectReader::new_with_header_builder(
- TestSingleObjectReader::get_schema(),
- header_builder,
- )
- .expect("failed to build reader");
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .header(header_builder.build_header())
+ .build()
+ .expect("failed to build reader");
let data_to_read: Vec<u8> = vec![
3, 0, 178, 241, 207, 0, 4, 52, 1, 62, 67, 154, 18, 94, 184, 72, 90, 95,
];
Review Comment:
The `matches!(read_result, Err(Details::ReadBytes(_)));` below should probably be wrapped with `assert!(...)`
GitHub UI does not allow me to add a suggestion
##########
avro/src/reader/block.rs:
##########
@@ -205,6 +215,44 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
Review Comment:
```suggestion
reader_schema: Option<&Schema>,
```
for consistency
##########
avro/src/schema/union.rs:
##########
@@ -74,7 +81,116 @@ impl UnionSchema {
&self.schemas
}
- /// Returns true if the any of the variants of this `UnionSchema` is `Null`.
+ /// Get the variant at the given index.
+ pub fn get_variant(&self, index: usize) -> Result<&Schema, Error> {
+ self.schemas.get(index).ok_or_else(|| {
+ Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.schemas.len(),
+ }
+ .into()
+ })
+ }
+
+ /// Get the index of the provided [`SchemaKind`].
+ ///
+ /// The schema must not be a logical type, as only the base type are saved in the lookup index.
+ pub(crate) fn index_of_schema_kind(&self, kind: SchemaKind) -> Option<usize> {
+ self.variant_index.get(&kind).copied()
+ }
+
+ /// Get the index and schema for the provided name.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_named_schema<'s>(
+ &'s self,
+ name: &str,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s Schema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ if let Some(schema_name) = schema.name()
+ && schema_name.name() == name
Review Comment:
Shouldn't this use the fully qualified names ?
##########
avro/src/serde/with.rs:
##########
@@ -503,11 +502,323 @@ pub mod slice_opt {
}
}
+/// (De)serialize [`BigDecimal`] as a [`Schema::BigDecimal`] instead of a [`Schema::String`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal_opt`] for optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal")]
+/// decimal: BigDecimal,
+/// }
+/// ```
+///
+/// [`BigDecimal`]: ::bigdecimal::BigDecimal
+/// [`Schema::BigDecimal`]: crate::Schema::BigDecimal
+/// [`Schema::String`]: crate::Schema::String
+/// [`apache_avro::serde::bigdecimal_opt`]: bigdecimal_opt
+pub mod bigdecimal {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField},
+ serde::with::BytesType,
+ };
+
+ /// Returns [`Schema::BigDecimal`]
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::BigDecimal
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &BigDecimal, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&decimal_bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<BigDecimal, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ // We don't use &'de [u8] here as the deserializer doesn't support that
+ let bytes: Vec<u8> = serde_bytes::deserialize(deserializer)?;
+
+ deserialize_big_decimal(&bytes).map_err(D::Error::custom)
+ }
+}
+
+/// (De)serialize [`Option<BigDecimal>`] as a `Schema::Union(Schema::Null, Schema::BigDecimal)` instead of a `Schema::Union(Schema::Null, Schema::String)`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal`] for non-optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal_opt")]
+/// decimal: Option<BigDecimal>,
+/// }
+/// ```
+///
+/// [`Option<BigDecimal>`]: ::bigdecimal::BigDecimal
+/// [`apache_avro::serde::bigdecimal`]: bigdecimal
+pub mod bigdecimal_opt {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField, UnionSchema},
+ serde::with::BytesType,
+ };
+
+ /// Returns `Schema::Union(Schema::Null, Schema::BigDecimal)`
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::Union(
+ UnionSchema::new(vec![Schema::Null, Schema::BigDecimal])
+ .unwrap_or_else(|_| unreachable!("This is a valid union")),
+ )
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &Option<BigDecimal>, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ if let Some(decimal) = decimal {
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&Some(decimal_bytes), serializer)
+ } else {
+ serde_bytes::serialize(&None::<Vec<u8>>, serializer)
+ }
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<BigDecimal>, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ let bytes: Option<Vec<u8>> = serde_bytes::deserialize(deserializer)?;
+ if let Some(bytes) = bytes {
+ deserialize_big_decimal(&bytes)
+ .map(Some)
+ .map_err(D::Error::custom)
+ } else {
+ Ok(None)
+ }
+ }
+}
+
+/// (De)serialize an Rust array (`[T; N]`) as an Avro [`Schema::Array`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array_opt`] for optional array values.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithArray {
+/// #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<i32>)]
+/// #[serde(with = "apache_avro::serde::array")]
+/// array: [i32; 10],
+/// }
+/// ```
+///
+/// [`apache_avro::serde::array_opt`]: array_opt
+/// [`Schema::Array`]: crate::schema::Schema::Array
+pub mod array {
+ use std::collections::HashSet;
+
+ use serde::{Deserialize, Deserializer, Serialize, Serializer, de::Error as _};
+
+ use crate::{
+ AvroSchemaComponent, Schema,
+ schema::{Name, NamespaceRef, RecordField},
+ };
+
+ /// Returns `Schema::Array(T::get_schema_in_ctxt())`
+ pub fn get_schema_in_ctxt<T: AvroSchemaComponent>(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build()
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<const N: usize, S, T>(value: &[T; N], serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ T: Serialize,
+ {
+ value.as_slice().serialize(serializer)
+ }
+
+ pub fn deserialize<'de, const N: usize, D, T>(deserializer: D) -> Result<[T; N], D::Error>
+ where
+ D: Deserializer<'de>,
+ T: Deserialize<'de>,
+ {
+ let bytes = <Vec<T> as Deserialize>::deserialize(deserializer)?;
+ bytes.try_into().map_err(|v: Vec<T>| {
+ D::Error::custom(format!(
+ "Deserialized array has length {} which does not match array length of {N}",
+ v.len()
+ ))
+ })
+ }
+}
+
+/// (De)serialize an optional Rust array (`Option<[T; N]>`) as an Avro `Schema::Union([Schema::Null, Schema::Array])`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array`] for non-optional array values.
+///
+/// When used with different serialization formats, this is equivalent to [`serde_bytes`].
Review Comment:
Is this copy/pasted ? It sounds unrelated here.
##########
avro/src/reader/single_object.rs:
##########
@@ -68,6 +79,26 @@ impl GenericSingleObjectReader {
Err(io_error) => Err(Details::ReadHeader(io_error).into()),
}
}
+
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ let mut header = vec![0; self.expected_header.len()];
Review Comment:
The header reading and validation logic is the same in `read_value()` and `read_deser()`. It could be extracted to a helper function
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](#enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](#enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema of the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples.
+//! - **tuple_variant** => See [Enums](#enums)
+//! - **map** => [`Schema::Map`]
+//! - **Note:** the key type of the map will be (de)serialized as a [`Schema::String`]
+//! - **struct** => [`Schema::Record`]
+//! - **struct_variant** => See [Enums](#enums)
+//!
+//! ## Enums
+//!
+//! ### Externally tagged
+//! This is the default enum representation for Serde. It can be mapped in three ways to the Avro data model.
+//! For all three options it is important that the enum index matches the Avro index.
+//! - As a [`Schema::Enum`], this is only possible for enums with only unit variants.
+//! - As a [`Schema::Union`] with a [`Schema::Record`] for every variant:
+//! - **unit_variant** => [`Schema::Record`] named as the variant and with no fields.
+//! - **newtype_variant** => [`Schema::Record`] named as the variant and with one field.
+//! The schema must have the attribute `org.apache.avro.rust.union_of_records` with the value set to `true`.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
Review Comment:
```suggestion
//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are elements.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2972082390
##########
avro/src/serde/deser_schema/record.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, MapAccess};
+
+use crate::{
+ Error, Schema,
+ schema::RecordSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer, identifier::IdentifierDeserializer},
+};
+
+pub struct RecordDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> RecordDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> MapAccess<'de> for RecordDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_key_seed<K>(&mut self, seed: K) -> Result<Option<K::Value>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ {
+ if self.current_field >= self.schema.fields.len() {
+ // Finished reading this record
+ Ok(None)
+ } else {
+ seed.deserialize(IdentifierDeserializer::string(
+ &self.schema.fields[self.current_field].name,
+ ))
+ .map(Some)
+ }
+ }
+
+ fn next_value_seed<V>(&mut self, seed: V) -> Result<V::Value, Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let schema = &self.schema.fields[self.current_field].schema;
Review Comment:
It is checked in `next_key_seed` and the Serde docs say we're allowed to panic if `next_value_seed` is called without `next_key_seed`: https://docs.rs/serde/latest/serde/de/trait.MapAccess.html#tymethod.next_value_seed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969300320
##########
avro/src/reader/datum.rs:
##########
@@ -124,6 +137,68 @@ impl<'s> GenericDatumReader<'s> {
Ok(value)
}
}
+
+ /// Read a Avro datum from the reader.
+ ///
+ /// # Panics
+ /// Will panic if a reader schema has been configured, this is a WIP.
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ // `reader` is `impl Read` instead of a generic on the function like T so it's easier to
+ // specify the type wanted (`read_deser<String>` vs `read_deser<String, _>`)
+ if let Some((_, _)) = &self.reader {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
Review Comment:
Follow-up, I've changed it to a panic and a comment. It's also documented as a panic on `read_deser`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2965512779
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the the mapping as defined here is supported. Any other behavior might change in a minor version.
Review Comment:
```suggestion
//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
```
##########
avro/src/schema/union.rs:
##########
@@ -74,7 +81,116 @@ impl UnionSchema {
&self.schemas
}
- /// Returns true if the any of the variants of this `UnionSchema` is `Null`.
+ /// Get the variant at the given index.
+ pub fn get_variant(&self, index: usize) -> Result<&Schema, Error> {
+ self.schemas.get(index).ok_or_else(|| {
+ Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.schemas.len(),
+ }
+ .into()
+ })
+ }
+
+ /// Get the index of the provided [`SchemaKind`].
+ ///
+ /// The schema must not be a logical type, as only the base type are saved in the lookup index.
+ pub(crate) fn index_of_schema_kind(&self, kind: SchemaKind) -> Option<usize> {
+ self.variant_index.get(&kind).copied()
+ }
+
+ /// Get the index and schema for the provided name.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_named_schema<'s>(
+ &'s self,
+ name: &str,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s Schema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ if let Some(schema_name) = schema.name()
+ && schema_name.name() == name
+ {
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ return Ok(Some((index, schema)));
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Fixed`] with the given size.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_fixed_of_size_n<'s>(
+ &'s self,
+ size: usize,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s FixedSchema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ match schema {
+ Schema::Fixed(fixed)
+ | Schema::Uuid(UuidSchema::Fixed(fixed))
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(fixed),
+ ..
+ })
+ | Schema::Duration(fixed)
+ if fixed.size == size =>
+ {
+ return Ok(Some((index, fixed)));
+ }
+ _ => {}
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Record`] with `n` fields.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_record_with_n_fields<'s>(
Review Comment:
What is the idea here ?
If a union schema has several record schemas with the same number of fields but with different types of these fields then the first one wins.
Why do we need this ?
##########
avro/src/reader/datum.rs:
##########
@@ -124,6 +137,68 @@ impl<'s> GenericDatumReader<'s> {
Ok(value)
}
}
+
+ /// Read a Avro datum from the reader.
+ ///
+ /// # Panics
+ /// Will panic if a reader schema has been configured, this is a WIP.
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ // `reader` is `impl Read` instead of a generic on the function like T so it's easier to
+ // specify the type wanted (`read_deser<String>` vs `read_deser<String, _>`)
+ if let Some((_, _)) = &self.reader {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
Review Comment:
Is this a todo for the current PR or for a follow-up ?
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
Review Comment:
```suggestion
(Value::Number(n), Schema::Float | Schema::Double) if n.as_f64().is_some() => true,
```
to support also integers as defaults for floating point schema types
##########
avro/src/serde/ser_schema/record/mod.rs:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod field_default;
+
+use std::{borrow::Borrow, cmp::Ordering, collections::HashMap, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeStruct, SerializeStructVariant},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::RecordSchema,
+ serde::{
+ ser_schema::record::field_default::SchemaAwareRecordFieldDefault, util::StringSerializer,
+ },
+};
+
+pub struct RecordSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ /// Cache fields received out-of-order
+ cache: HashMap<usize, Vec<u8>>,
+ /// The position of the current map entry being written
+ map_position: Option<usize>,
+ /// The field that should be written now.
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ record,
+ config,
+ cache: HashMap::new(),
+ map_position: None,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+
+ fn field_error(&self, position: usize, error: Error) -> Error {
+ let field = &self.record.fields[position];
+ let error = match error.into_details() {
+ Details::SerializeValueWithSchema {
+ value_type,
+ value,
+ schema: _,
+ } => format!("Failed to serialize value of type `{value_type}`: {value}"),
+ Details::SerializeRecordFieldWithSchema {
+ field_name,
+ record_schema,
+ error,
+ } => format!(
+ "Failed to serialize field '{field_name}' of record {}: {error}",
+ record_schema.name
+ ),
+ Details::MissingDefaultForSkippedField { field_name, schema } => {
+ format!(
+ "Missing default for skipped field '{field_name}' for record {}",
+ schema.name
+ )
+ }
+ details => format!("{details:?}"),
+ };
+ Error::new(Details::SerializeRecordFieldWithSchema {
+ field_name: field.name.clone(),
+ record_schema: self.record.clone(),
+ error,
+ })
+ }
+
+ fn serialize_next_field<T: ?Sized + Serialize>(
+ &mut self,
+ position: usize,
+ value: &T,
+ ) -> Result<(), Error> {
+ let field = &self.record.fields[position];
+ match self.field_position.cmp(&position) {
+ Ordering::Equal => {
+ // Field received in the right order
+ self.bytes_written += value
+ .serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &field.schema,
+ self.config,
+ )?)
+ .map_err(|e| self.field_error(self.field_position, e))?;
+ self.field_position += 1;
+
+ // Write any fields that were already received and can now be written
+ while let Some(bytes) = self.cache.remove(&self.field_position) {
+ self.writer.write_all(&bytes).map_err(Details::WriteBytes)?;
+ self.bytes_written += bytes.len();
+ self.field_position += 1;
+ }
+
+ Ok(())
+ }
+ Ordering::Less => {
+ // Another field needs to be written first, so cache this field
+ let mut bytes = Vec::new();
+ value
+ .serialize(SchemaAwareSerializer::new(
+ &mut bytes,
+ &field.schema,
+ self.config,
+ )?)
+ .map_err(|e| self.field_error(self.field_position, e))?;
+ if self.cache.insert(position, bytes).is_some() {
+ Err(Details::FieldNameDuplicate(field.name.clone()).into())
+ } else {
+ Ok(())
+ }
+ }
+ Ordering::Greater => {
+ // This field is already written to the writer so we got a duplicate
+ Err(Details::FieldNameDuplicate(field.name.clone()).into())
+ }
+ }
+ }
+
+ fn serialize_default(&mut self, position: usize) -> Result<(), Error> {
+ let field = &self.record.fields[position];
+ if let Some(default) = &field.default {
+ self.serialize_next_field(
+ position,
+ &SchemaAwareRecordFieldDefault::new(default, &field.schema),
+ )
+ .map_err(|e| self.field_error(position, e))
+ } else {
+ Err(Details::MissingDefaultForSkippedField {
+ field_name: field.name.clone(),
+ schema: self.record.clone(),
+ }
+ .into())
+ }
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any fields that were skipped by `#[serde(skip)]` or `#[serde(skip_serializing{,_if}]`
+ while self.field_position != self.record.fields.len() {
+ self.serialize_default(self.field_position)?;
+ }
+
+ Ok(self.bytes_written)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeStruct for RecordSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ if let Some(position) = self.record.lookup.get(key).copied() {
+ self.serialize_next_field(position, value)
+ } else {
+ Err(Details::GetField(key.to_string()).into())
+ }
+ }
+
+ fn skip_field(&mut self, key: &'static str) -> Result<(), Self::Error> {
+ if let Some(position) = self.record.lookup.get(key).copied() {
+ self.serialize_default(position)
+ } else {
+ Err(Details::GetField(key.to_string()).into())
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for RecordSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let name = key.serialize(StringSerializer)?;
+ if let Some(position) = self.record.lookup.get(&name).copied() {
+ self.map_position = Some(position);
+ Ok(())
+ } else {
+ Err(Details::FieldName(name.to_string()).into())
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_next_field(
+ self.map_position
+ .expect("serialze_value called without calling serialize_key"),
Review Comment:
```suggestion
.expect("serialize_value called without calling serialize_key"),
```
##########
avro/src/writer/single_object.rs:
##########
@@ -86,12 +89,35 @@ impl GenericSingleObjectWriter {
}
/// Writer that encodes messages according to the single object encoding v1 spec
+#[derive(Builder)]
pub struct SpecificSingleObjectWriter<T>
where
T: AvroSchema,
{
+ #[builder(
+ with = |schema: Schema| -> Result<_, Error> { ResolvedOwnedSchema::new(schema) },
+ default = ResolvedOwnedSchema::new(T::get_schema()).expect("AvroSchema implementation should create valid schemas")
+ )]
resolved: ResolvedOwnedSchema,
+ #[builder(
+ default = RabinFingerprintHeader::from_schema(resolved.get_root_schema()).build_header(),
+ with = |header_builder: impl HeaderBuilder| header_builder.build_header(),
+ )]
header: Vec<u8>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
Review Comment:
```suggestion
/// Should [`Serialize`] implementations pick a human readable representation.
```
##########
avro/src/serde/ser_schema/tuple.rs:
##########
@@ -0,0 +1,283 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeTuple, SerializeTupleStruct, SerializeTupleVariant},
+};
+
+use super::{Config, SchemaAwareSerializer, union::UnionSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{RecordSchema, UnionSchema},
+};
+
+#[expect(
+ private_interfaces,
+ reason = "One{,Union}TupleSerializer should not be used directly"
+)]
+pub enum TupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Unit(usize),
+ One(OneTupleSerializer<'s, 'w, W, S>),
+ /// This exists because we can't create a `&Schema::Union` from a `&UnionSchema`
+ OneUnion(OneUnionTupleSerializer<'s, 'w, W, S>),
+ Many(ManyTupleSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> TupleSerializer<'s, 'w, W, S> {
+ pub fn unit(bytes_written: Option<usize>) -> Self {
+ Self::Unit(bytes_written.unwrap_or(0))
+ }
+
+ pub fn one(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::One(OneTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+
+ pub fn one_union(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self::OneUnion(OneUnionTupleSerializer::new(writer, union, config))
+ }
+
+ pub fn many(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::Many(ManyTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for TupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ TupleSerializer::Unit(_) => Err(Error::new(Details::SerializeValueWithSchema {
+ value_type: "tuple",
+ value: "Expected no elements for the unit tuple".into(),
+ schema: Schema::Null,
+ })),
+ TupleSerializer::One(one) => one.serialize_element(value),
+ TupleSerializer::OneUnion(one) => one.serialize_element(value),
+ TupleSerializer::Many(many) => many.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ TupleSerializer::Unit(bytes_written) => Ok(bytes_written),
+ TupleSerializer::One(one) => one.end(),
+ TupleSerializer::OneUnion(one) => one.end(),
+ TupleSerializer::Many(many) => SerializeTuple::end(many),
+ }
+ }
+}
+
+pub struct ManyTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> ManyTupleSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ schema,
+ config,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for ManyTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let schema = &self.schema.fields[self.field_position].schema;
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ schema,
+ self.config,
+ )?)?;
+ self.field_position += 1;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ assert_eq!(self.field_position, self.schema.fields.len());
Review Comment:
Why assert_eq!(), i.e. a possible panic ?
Why not an Err or at least a debug_assert_eq!()?
##########
avro/src/reader/mod.rs:
##########
@@ -113,6 +125,10 @@ impl<'a, R: Read> Reader<'a, R> {
self.block.read_next(read_schema)
}
+
+ fn read_next_deser<T: DeserializeOwned>(&mut self) -> AvroResult<Option<T>> {
+ self.block.read_next_deser(self.reader_schema)
Review Comment:
Should this also use `if self.should_resolve_schema {}` as ::read_next() does above ?
##########
avro/src/reader/block.rs:
##########
@@ -205,6 +215,43 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
+ ) -> AvroResult<Option<T>> {
+ if self.is_empty() {
+ self.read_block_next()?;
+ if self.is_empty() {
+ return Ok(None);
+ }
+ }
+
+ let mut block_bytes = &self.buf[self.buf_idx..];
+ let b_original = block_bytes.len();
+
+ let item = if read_schema.is_some() {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
Review Comment:
Is this a todo for the current PR or for a follow-up ?
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol most match the index of the variant
Review Comment:
```suggestion
//! - The index of the symbol must match the index of the variant
```
##########
avro/src/serde/mod.rs:
##########
@@ -27,7 +27,8 @@
//! details on how to change the generated schema.
//!
//! Alternatively, you can write your own schema. If you go down this path, it is recommended you start with
-//! the schema derived by [`AvroSchema`] and then modify it to fit your needs.
+//! the schema derived by [`AvroSchema`] and then modify it to fit your needs. For more information on mapping
+//! between Sere and Avro see [Avro to Serde](crate::documentation::avro_data_model_to_serde) and [Serde to Avro](crate::documentation::serde_data_model_to_avro).
Review Comment:
```suggestion
//! between Serde and Avro see [Avro to Serde](crate::documentation::avro_data_model_to_serde) and [Serde to Avro](crate::documentation::serde_data_model_to_avro).
```
##########
avro/src/serde/ser_schema/block.rs:
##########
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeSeq},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ encode::encode_int,
+ error::Details,
+ schema::{ArraySchema, MapSchema},
+ util::zig_i32,
+};
+
+#[expect(
+ private_interfaces,
+ reason = "Direct and Buffered should not be used directly"
+)]
+pub enum BlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Direct(DirectBlockSerializer<'s, 'w, W, S>),
+ Buffered(BufferedBlockSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BlockSerializer<'s, 'w, W, S> {
+ pub fn array(
+ writer: &'w mut W,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let bytes_written = bytes_written.unwrap_or(0);
+ if let Some(len) = len
+ && config.target_block_size.is_none()
+ {
+ Ok(Self::Direct(DirectBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ } else {
+ let target_block_size = config.target_block_size.unwrap_or(1024);
+ Ok(Self::Buffered(BufferedBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ target_block_size,
+ bytes_written,
+ )))
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_element(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_key(key),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_value(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+struct DirectBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> DirectBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: usize,
+ mut bytes_written: usize,
+ ) -> Result<Self, Error> {
+ if len != 0 {
+ // .end() always writes the zero block, so we only write the size for arrays
+ // that have at least one element
+ bytes_written += zig_i32(len as i32, &mut *writer)?;
+ }
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ bytes_written,
+ })
+ }
+
+ fn end(self) -> Result<usize, Error> {
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+struct BufferedBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ buffer: Vec<u8>,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+ items_in_buffer: usize,
+ target_block_size: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BufferedBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ target_block_size: usize,
+ bytes_written: usize,
+ ) -> Self {
+ Self {
+ writer,
+ buffer: Vec::with_capacity(target_block_size),
+ schema,
+ config,
+ bytes_written,
+ items_in_buffer: 0,
+ target_block_size,
+ }
+ }
+
+ /// Write a block including the items and size header.
+ fn write_block(&mut self) -> Result<(), Error> {
+ // Write the header, the negative item count indicates that the next value is the size of the
+ // block in bytes
+ self.bytes_written += encode_int(0 - (self.items_in_buffer as i32), &mut *self.writer)?;
+ self.bytes_written += encode_int(self.buffer.len() as i32, &mut *self.writer)?;
+
+ // Write the actual data
+ self.writer
+ .write_all(&self.buffer)
+ .map_err(Details::WriteBytes)?;
+ self.bytes_written += self.buffer.len();
+
+ // Reset the buffer
+ self.items_in_buffer = 0;
+ self.buffer.clear();
+
+ Ok(())
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any items remaining in the buffer
+ if self.items_in_buffer > 0 {
+ self.write_block()?;
+ }
+ debug_assert_eq!(self.buffer.len(), 0, "Buffer must be empty at this point");
+
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ value.serialize(SchemaAwareSerializer::new(
+ self.writer,
Review Comment:
Shouldn't this use the buffer ?
```suggestion
&mut self.buffer,
```
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema the only element
Review Comment:
```suggestion
//! - For tuples with one element, the schema will be the schema of the only element
```
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
Review Comment:
this seems to point to the current module/document
##########
avro/src/schema/mod.rs:
##########
@@ -784,6 +785,69 @@ impl Schema {
}
Ok(())
}
+
+ /// Derive a name for this schema.
+ ///
+ /// The name is a valid schema name and will be unique if the named
+ /// schemas in this schema have unique names.
+ pub(crate) fn unique_normalized_name(&self) -> Cow<'static, str> {
+ match self {
+ Schema::Null => Cow::Borrowed("n"),
+ Schema::Boolean => Cow::Borrowed("B"),
+ Schema::Int => Cow::Borrowed("i"),
+ Schema::Long => Cow::Borrowed("l"),
+ Schema::Float => Cow::Borrowed("f"),
+ Schema::Double => Cow::Borrowed("d"),
+ Schema::Bytes => Cow::Borrowed("b"),
+ Schema::String => Cow::Borrowed("s"),
+ Schema::Array(array) => {
+ Cow::Owned(format!("a_{}", array.items.unique_normalized_name()))
+ }
+ Schema::Map(map) => Cow::Owned(format!("m_{}", map.types.unique_normalized_name())),
+ Schema::Union(union) => {
+ let mut name = format!("u{}", union.schemas.len());
+ for schema in &union.schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ Cow::Owned(name)
+ }
+ Schema::BigDecimal => Cow::Borrowed("bd"),
+ Schema::Date => Cow::Borrowed("D"),
+ Schema::TimeMillis => Cow::Borrowed("t"),
+ Schema::TimeMicros => Cow::Borrowed("tm"),
+ Schema::TimestampMillis => Cow::Borrowed("T"),
+ Schema::TimestampMicros => Cow::Borrowed("TM"),
+ Schema::TimestampNanos => Cow::Borrowed("TN"),
+ Schema::LocalTimestampMillis => Cow::Borrowed("L"),
+ Schema::LocalTimestampMicros => Cow::Borrowed("LM"),
+ Schema::LocalTimestampNanos => Cow::Borrowed("LN"),
+ Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ precision,
+ scale,
+ }) => Cow::Owned(format!("db_{precision}_{scale}")),
+ Schema::Uuid(UuidSchema::Bytes) => Cow::Borrowed("ub"),
+ Schema::Uuid(UuidSchema::String) => Cow::Borrowed("us"),
+ Schema::Record(RecordSchema { name, .. })
+ | Schema::Enum(EnumSchema { name, .. })
+ | Schema::Fixed(FixedSchema { name, .. })
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(FixedSchema { name, .. }),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(FixedSchema { name, .. }))
+ | Schema::Duration(FixedSchema { name, .. })
+ | Schema::Ref { name } => {
+ let name: String = name
+ .to_string()
+ .chars()
+ .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
Review Comment:
IMO `_` should be cared specially here.
Currently two distinct names as `a.b.c` and `a_b.c` will lead to `r5_a_b_c`.
##########
avro/src/serde/deser_schema/tuple.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, SeqAccess};
+
+use crate::{
+ Error, Schema,
+ schema::RecordSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer},
+};
+
+pub struct OneTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ field_read: bool,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> OneTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema {
+ config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ field_read: false,
+ })
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for OneTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.field_read {
+ Ok(None)
+ } else {
+ self.field_read = true;
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)
+ .map(Some)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ Some(1 - usize::from(self.field_read))
+ }
+}
+
+pub struct ManyTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> ManyTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for ManyTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.current_field < self.schema.fields.len() {
+ let schema = &self.schema.fields[self.current_field].schema;
+ self.current_field += 1;
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ schema,
+ self.config,
+ )?)
+ .map(Some)
+ } else {
+ Ok(None)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ todo!()
Review Comment:
```suggestion
Some(self.schema.fields.len().saturating_sub(self.current_field))
```
##########
avro/src/serde/ser_schema/block.rs:
##########
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeSeq},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ encode::encode_int,
+ error::Details,
+ schema::{ArraySchema, MapSchema},
+ util::zig_i32,
+};
+
+#[expect(
+ private_interfaces,
+ reason = "Direct and Buffered should not be used directly"
+)]
+pub enum BlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Direct(DirectBlockSerializer<'s, 'w, W, S>),
+ Buffered(BufferedBlockSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BlockSerializer<'s, 'w, W, S> {
+ pub fn array(
+ writer: &'w mut W,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let bytes_written = bytes_written.unwrap_or(0);
+ if let Some(len) = len
+ && config.target_block_size.is_none()
+ {
+ Ok(Self::Direct(DirectBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ } else {
+ let target_block_size = config.target_block_size.unwrap_or(1024);
+ Ok(Self::Buffered(BufferedBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ target_block_size,
+ bytes_written,
+ )))
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_element(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_key(key),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_value(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+struct DirectBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> DirectBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: usize,
+ mut bytes_written: usize,
+ ) -> Result<Self, Error> {
+ if len != 0 {
+ // .end() always writes the zero block, so we only write the size for arrays
+ // that have at least one element
+ bytes_written += zig_i32(len as i32, &mut *writer)?;
+ }
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ bytes_written,
+ })
+ }
+
+ fn end(self) -> Result<usize, Error> {
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+struct BufferedBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ buffer: Vec<u8>,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+ items_in_buffer: usize,
+ target_block_size: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BufferedBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ target_block_size: usize,
+ bytes_written: usize,
+ ) -> Self {
+ Self {
+ writer,
+ buffer: Vec::with_capacity(target_block_size),
+ schema,
+ config,
+ bytes_written,
+ items_in_buffer: 0,
+ target_block_size,
+ }
+ }
+
+ /// Write a block including the items and size header.
+ fn write_block(&mut self) -> Result<(), Error> {
+ // Write the header, the negative item count indicates that the next value is the size of the
+ // block in bytes
+ self.bytes_written += encode_int(0 - (self.items_in_buffer as i32), &mut *self.writer)?;
+ self.bytes_written += encode_int(self.buffer.len() as i32, &mut *self.writer)?;
+
+ // Write the actual data
+ self.writer
+ .write_all(&self.buffer)
+ .map_err(Details::WriteBytes)?;
+ self.bytes_written += self.buffer.len();
+
+ // Reset the buffer
+ self.items_in_buffer = 0;
+ self.buffer.clear();
+
+ Ok(())
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any items remaining in the buffer
+ if self.items_in_buffer > 0 {
+ self.write_block()?;
+ }
+ debug_assert_eq!(self.buffer.len(), 0, "Buffer must be empty at this point");
+
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ self.items_in_buffer += 1;
+
+ if self.buffer.len() >= self.target_block_size {
+ self.write_block()?;
+ }
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ key.serialize(SchemaAwareSerializer::new(
+ self.writer,
Review Comment:
Shouldn't this use the buffer ?
```suggestion
&mut self.buffer,
```
##########
avro/src/error.rs:
##########
@@ -156,7 +157,6 @@ pub enum Details {
#[error("Union index {index} out of bounds: {num_variants}")]
GetUnionVariant { index: i64, num_variants: usize },
- #[deprecated(since = "0.20.0", note = "This error variant is not generated anymore")]
#[error("Enum symbol index out of bounds: {num_variants}")]
Review Comment:
let's add `index` to the error message
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.is_f64() => {
Review Comment:
```suggestion
(Value::Number(n), Schema::Float) if n.as_f64().is_some() => {
```
##########
avro/src/serde/deser_schema/map.rs:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, MapAccess};
+
+use crate::{
+ Error, Schema,
+ schema::MapSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer},
+ util::{zag_i32, zag_i64},
+};
+
+pub struct MapDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ remaining: Option<u32>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> MapDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+ let remaining = Self::read_block_header(reader)?;
+ Ok(Self {
+ schema,
+ reader,
+ config,
+ remaining,
+ })
+ }
+
+ fn read_block_header(reader: &mut R) -> Result<Option<u32>, Error> {
+ let remaining = zag_i32(reader)?;
+ if remaining < 0 {
+ // If the block size is negative the next number is the size of the block in bytes
+ let _bytes = zag_i64(reader)?;
+ }
+ if remaining == 0 {
+ // If the block size is zero the array is finished
Review Comment:
```suggestion
// If the block size is zero the map is finished
```
##########
avro/src/writer/mod.rs:
##########
@@ -67,6 +70,19 @@ impl<'a, W: Write> Writer<'a, W> {
#[builder(default = false)]
has_header: bool,
#[builder(default)] user_metadata: HashMap<String, Value>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
Review Comment:
```suggestion
/// Should [`Serialize`] implementations pick a human readable representation.
```
##########
avro/src/reader/block.rs:
##########
@@ -205,6 +215,43 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
+ ) -> AvroResult<Option<T>> {
+ if self.is_empty() {
+ self.read_block_next()?;
+ if self.is_empty() {
+ return Ok(None);
+ }
+ }
+
+ let mut block_bytes = &self.buf[self.buf_idx..];
+ let b_original = block_bytes.len();
+
+ let item = if read_schema.is_some() {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
+ } else {
+ let config = Config {
+ names: &self.names_refs,
+ human_readable: self.human_readable,
+ };
+ T::deserialize(SchemaAwareDeserializer::new(
+ &mut block_bytes,
+ &self.writer_schema,
+ config,
+ )?)?
+ };
+
+ if b_original != 0 && b_original == block_bytes.len() {
+ // No bytes were read, return an error to avoid an infinite loop
+ return Err(Details::ReadBlock.into());
Review Comment:
Let's add some more context information to this error.
As a user I won't be sure what to do.
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
Review Comment:
What about the logical types with inner type, e.g. Uuid, Decimal & Duration) ?
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1688 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod array;
+mod enums;
+mod identifier;
+mod map;
+mod record;
+mod tuple;
+
+use array::ArrayDeserializer;
+use enums::PlainEnumDeserializer;
+use map::MapDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let variant = schema.get_variant(index as usize)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
Review Comment:
```suggestion
Schema::Union(union) => self.with_union(union)?.deserialize_u128(visitor),
```
copy/paste error ?
##########
avro/src/serde/ser_schema/record/mod.rs:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod field_default;
+
+use std::{borrow::Borrow, cmp::Ordering, collections::HashMap, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeStruct, SerializeStructVariant},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::RecordSchema,
+ serde::{
+ ser_schema::record::field_default::SchemaAwareRecordFieldDefault, util::StringSerializer,
+ },
+};
+
+pub struct RecordSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ /// Cache fields received out-of-order
+ cache: HashMap<usize, Vec<u8>>,
+ /// The position of the current map entry being written
+ map_position: Option<usize>,
+ /// The field that should be written now.
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ record,
+ config,
+ cache: HashMap::new(),
+ map_position: None,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+
+ fn field_error(&self, position: usize, error: Error) -> Error {
+ let field = &self.record.fields[position];
+ let error = match error.into_details() {
+ Details::SerializeValueWithSchema {
+ value_type,
+ value,
+ schema: _,
+ } => format!("Failed to serialize value of type `{value_type}`: {value}"),
+ Details::SerializeRecordFieldWithSchema {
+ field_name,
+ record_schema,
+ error,
+ } => format!(
+ "Failed to serialize field '{field_name}' of record {}: {error}",
+ record_schema.name
+ ),
+ Details::MissingDefaultForSkippedField { field_name, schema } => {
+ format!(
+ "Missing default for skipped field '{field_name}' for record {}",
+ schema.name
+ )
+ }
+ details => format!("{details:?}"),
+ };
+ Error::new(Details::SerializeRecordFieldWithSchema {
+ field_name: field.name.clone(),
+ record_schema: self.record.clone(),
+ error,
+ })
+ }
+
+ fn serialize_next_field<T: ?Sized + Serialize>(
+ &mut self,
+ position: usize,
+ value: &T,
+ ) -> Result<(), Error> {
+ let field = &self.record.fields[position];
+ match self.field_position.cmp(&position) {
+ Ordering::Equal => {
+ // Field received in the right order
+ self.bytes_written += value
+ .serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &field.schema,
+ self.config,
+ )?)
+ .map_err(|e| self.field_error(self.field_position, e))?;
+ self.field_position += 1;
+
+ // Write any fields that were already received and can now be written
+ while let Some(bytes) = self.cache.remove(&self.field_position) {
+ self.writer.write_all(&bytes).map_err(Details::WriteBytes)?;
+ self.bytes_written += bytes.len();
+ self.field_position += 1;
+ }
+
+ Ok(())
+ }
+ Ordering::Less => {
+ // Another field needs to be written first, so cache this field
+ let mut bytes = Vec::new();
+ value
+ .serialize(SchemaAwareSerializer::new(
+ &mut bytes,
+ &field.schema,
+ self.config,
+ )?)
+ .map_err(|e| self.field_error(self.field_position, e))?;
Review Comment:
```suggestion
.map_err(|e| self.field_error(position, e))?;
```
Use the requested position for the error, not the other field's position
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
Review Comment:
```suggestion
Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
```
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.is_f64() => {
+ let float = n.as_f64().unwrap();
+ serializer.serialize_f32(float as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.is_f64() => {
Review Comment:
```suggestion
(Value::Number(n), Schema::Double) if n.as_f64().is_some() => {
```
##########
avro/src/writer/datum.rs:
##########
@@ -59,6 +62,19 @@ impl<'s> GenericDatumWriter<'s> {
/// written data unreadable.
#[builder(default = true)]
validate: bool,
+ /// At what block size to start a new block (for arrays and maps).
+ ///
+ /// This is a minimum value, the block size will always be larger than this except for the last
+ /// block.
+ ///
+ /// When set to `None` all values will be written in a single block. This can be faster as no
+ /// intermediate buffer is used, but seeking through written data will be slower.
+ target_block_size: Option<usize>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
Review Comment:
```suggestion
/// Should [`Serialize`] implementations pick a human readable representation.
```
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.is_f64() => {
+ let float = n.as_f64().unwrap();
+ serializer.serialize_f32(float as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.is_f64() => {
+ let double = n.as_f64().unwrap();
+ serializer.serialize_f64(double)
+ }
+ (
+ Value::String(s),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_))
+ | Schema::BigDecimal
+ | Schema::Decimal(_)
+ | Schema::Duration(_)
+ | Schema::Date,
+ ) => serializer.serialize_bytes(s.as_bytes()),
+ (Value::String(s), Schema::String | Schema::Uuid(UuidSchema::String)) => {
+ serializer.serialize_str(s)
+ }
+ (Value::String(s), Schema::Enum(enum_schema)) => {
+ let Some((variant_index, _)) = enum_schema
+ .symbols
+ .iter()
+ .enumerate()
+ .find(|(_i, symbol)| *symbol == s)
+ else {
+ return Err(S::Error::custom(format!(
+ "Could not find `{s}` in enum: {enum_schema:?}"
+ )));
+ };
+
+ serializer.serialize_unit_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ variant_index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ )
+ }
+ // This abuses the support for flattened fields, which are also serialized as a map.
+ (Value::Object(o), Schema::Record(record)) => serializer.collect_map(
+ o.iter()
+ .enumerate()
+ .map(|(i, (k, v))| (k, Self::new(v, &record.fields[i].schema))),
+ ),
Review Comment:
```suggestion
(Value::Object(o), Schema::Record(record)) => serializer.collect_map(
record.fields.iter().filter_map(|field| {
o.get(&field.name)
.map(|value| (&field.name, Self::new(value, &field.schema)))
}),
),
```
to serialize the defaults in the schema order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2980265807
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1817 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod block;
+mod enums;
+mod identifier;
+mod record;
+mod tuple;
+
+use block::BlockDeserializer;
+use enums::PlainEnumDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let variant = schema.get_variant(index)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u128(visitor),
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_f32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Float => visitor.visit_f32(f32::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f32(visitor),
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
+ }
+
+ fn deserialize_f64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Double => visitor.visit_f64(f64::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f64(visitor),
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
+ }
+
+ fn deserialize_char<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // A char cannot be deserialized using Schema::Uuid(UuidSchema::String) as that is at least
+ // 32 characters.
+ Schema::String => {
+ let string = self.read_string()?;
+ let mut chars = string.chars();
+ let char = chars
+ .next()
+ .ok_or_else(|| self.error("char", "String is empty"))?;
+ // We can't just check the string length, as that is in bytes not characters
+ if chars.next().is_some() {
+ Err(self.error(
+ "char",
+ format!(r#"Read more than one character: "{string}""#),
+ ))
+ } else {
+ visitor.visit_char(char)
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_char(visitor),
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_str<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_string(visitor)
+ }
+
+ fn deserialize_string<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ visitor.visit_string(self.read_string()?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_string(visitor),
+ _ => Err(self.error("string", "Expected Schema::String | Schema::Uuid(String)")),
+ }
+ }
+
+ fn deserialize_bytes<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_byte_buf(visitor)
+ }
+
+ fn deserialize_byte_buf<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ visitor.visit_byte_buf(self.read_bytes_with_len()?)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), ..}) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ visitor.visit_byte_buf(self.read_bytes(fixed.size)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_byte_buf(visitor),
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Fixed | Bytes) | Schema::Duration")),
+ }
+ }
+
+ fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && union.is_nullable()
+ {
+ let index = zag_i32(self.reader)?;
+ let schema = union.get_variant(index as usize)?;
+ if let Schema::Null = schema {
+ visitor.visit_none()
+ } else {
+ visitor.visit_some(self.with_different_schema(schema)?)
+ }
+ } else {
+ Err(self.error("option", "Expected Schema::Union([Schema::Null, _])"))
+ }
+ }
+
+ fn deserialize_unit<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => visitor.visit_unit(),
+ Schema::Union(union) => self.with_union(union)?.deserialize_unit(visitor),
+ _ => Err(self.error("unit", "Expected Schema::Null")),
+ }
+ }
+
+ fn deserialize_unit_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ visitor.visit_unit()
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_unit_struct(name, visitor),
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 0)"),
+ )),
+ }
+ }
+
+ fn deserialize_newtype_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == 1 && record.name.name() == name => {
+ visitor.visit_newtype_struct(self.with_different_schema(&record.fields[0].schema)?)
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_newtype_struct(name, visitor),
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 1)"),
+ )),
+ }
+ }
+
+ fn deserialize_seq<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Array(array) => {
+ visitor.visit_seq(BlockDeserializer::array(self.reader, array, self.config)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_seq(visitor),
+ _ => Err(self.error("seq", "Expected Schema::Array")),
+ }
+ }
+
+ fn deserialize_tuple<V>(self, len: usize, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // `len == 0` is not possible for derived Deserialize implementations but users might use it.
+ // The derived Deserialize implementations use `deserialize_unit` instead
+ Schema::Null if len == 0 => visitor.visit_unit(),
+ schema if len == 1 => {
+ visitor.visit_seq(OneTupleDeserializer::new(self.reader, schema, self.config)?)
+ }
+ Schema::Record(record) if record.fields.len() == len => {
+ visitor.visit_seq(ManyTupleDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_tuple(len, visitor),
+ _ if len == 0 => Err(self.error("tuple", "Expected Schema::Null for unit tuple")),
+ _ => Err(self.error(
+ "tuple",
+ format!("Expected Schema::Record(fields.len() == {len}) for {len}-tuple"),
+ )),
+ }
+ }
+
+ fn deserialize_tuple_struct<V>(
+ self,
+ name: &'static str,
+ len: usize,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.name.name() == name && record.fields.len() == len => {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
Review Comment:
Yes, it shouldn't even use RecordDeserializer but ManyTupleDeserializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 merged PR #512:
URL: https://github.com/apache/avro-rs/pull/512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2972093864
##########
avro/src/error.rs:
##########
@@ -553,32 +555,51 @@ pub enum Details {
#[error("Decoded integer out of range for i32: {1}: {0}")]
ZagI32(#[source] std::num::TryFromIntError, i64),
- #[error("unable to read block")]
+ #[error("Did not read any bytes, block is corrupt")]
ReadBlock,
#[error("Failed to serialize value into Avro value: {0}")]
SerializeValue(String),
- #[error("Failed to serialize value of type {value_type} using schema {schema:?}: {value}")]
+ #[error("Failed to serialize value of type `{value_type}` using Schema::{schema:?}: {value}")]
SerializeValueWithSchema {
value_type: &'static str,
value: String,
schema: Schema,
},
- #[error("Failed to serialize field '{field_name}' for record {record_schema:?}: {error}")]
+ #[error("{position} is not a valid index for fields in {schema:?}")]
+ SerializeRecordUnknownFieldIndex {
+ position: usize,
+ schema: RecordSchema,
+ },
+
+ #[error("Failed to serialize field '{field_name}' of record {record_schema:?}: {error}")]
SerializeRecordFieldWithSchema {
field_name: String,
- record_schema: Schema,
- error: Box<Error>,
+ record_schema: RecordSchema,
+ error: String,
Review Comment:
So I can create a more optimal error string. When an error happens in the following chain: record -> field -> record -> field. The schema will be printed 4 times. This makes the error message almost useless. My modified error (as created in `RecordSerializer::field_error`) will only print the top schema and then just print the actual errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2972095813
##########
avro_derive/src/attributes/mod.rs:
##########
@@ -340,6 +340,12 @@ impl FieldOptions {
return Err(errors);
}
+ // TODO: Implement a better way to do this (maybe if user specifies `#[avro(with)]` also use that for the default)
+ // Disable getting the field default, if the schema is not retrieved from the field type
+ if with != With::Trait && avro.default == FieldDefault::Trait {
+ avro.default = FieldDefault::Disabled;
Review Comment:
Yes, I also have a better check in the next PR (with the support for deriving schemas for all enums)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4107011048
Map and Array deserialization now share a Block deserializer, just like in the serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3032071827
##########
avro/src/serde/with.rs:
##########
@@ -503,11 +502,323 @@ pub mod slice_opt {
}
}
+/// (De)serialize [`BigDecimal`] as a [`Schema::BigDecimal`] instead of a [`Schema::String`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal_opt`] for optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal")]
+/// decimal: BigDecimal,
+/// }
+/// ```
+///
+/// [`BigDecimal`]: ::bigdecimal::BigDecimal
+/// [`Schema::BigDecimal`]: crate::Schema::BigDecimal
+/// [`Schema::String`]: crate::Schema::String
+/// [`apache_avro::serde::bigdecimal_opt`]: bigdecimal_opt
+pub mod bigdecimal {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField},
+ serde::with::BytesType,
+ };
+
+ /// Returns [`Schema::BigDecimal`]
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::BigDecimal
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &BigDecimal, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&decimal_bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<BigDecimal, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ // We don't use &'de [u8] here as the deserializer doesn't support that
+ let bytes: Vec<u8> = serde_bytes::deserialize(deserializer)?;
+
+ deserialize_big_decimal(&bytes).map_err(D::Error::custom)
+ }
+}
+
+/// (De)serialize [`Option<BigDecimal>`] as a `Schema::Union(Schema::Null, Schema::BigDecimal)` instead of a `Schema::Union(Schema::Null, Schema::String)`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal`] for non-optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal_opt")]
+/// decimal: Option<BigDecimal>,
+/// }
+/// ```
+///
+/// [`Option<BigDecimal>`]: ::bigdecimal::BigDecimal
+/// [`apache_avro::serde::bigdecimal`]: bigdecimal
+pub mod bigdecimal_opt {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField, UnionSchema},
+ serde::with::BytesType,
+ };
+
+ /// Returns `Schema::Union(Schema::Null, Schema::BigDecimal)`
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::Union(
+ UnionSchema::new(vec![Schema::Null, Schema::BigDecimal])
+ .unwrap_or_else(|_| unreachable!("This is a valid union")),
+ )
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &Option<BigDecimal>, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ if let Some(decimal) = decimal {
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&Some(decimal_bytes), serializer)
+ } else {
+ serde_bytes::serialize(&None::<Vec<u8>>, serializer)
+ }
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<BigDecimal>, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ let bytes: Option<Vec<u8>> = serde_bytes::deserialize(deserializer)?;
+ if let Some(bytes) = bytes {
+ deserialize_big_decimal(&bytes)
+ .map(Some)
+ .map_err(D::Error::custom)
+ } else {
+ Ok(None)
+ }
+ }
+}
+
+/// (De)serialize an Rust array (`[T; N]`) as an Avro [`Schema::Array`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array_opt`] for optional array values.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithArray {
+/// #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<i32>)]
+/// #[serde(with = "apache_avro::serde::array")]
+/// array: [i32; 10],
+/// }
+/// ```
+///
+/// [`apache_avro::serde::array_opt`]: array_opt
+/// [`Schema::Array`]: crate::schema::Schema::Array
+pub mod array {
+ use std::collections::HashSet;
+
+ use serde::{Deserialize, Deserializer, Serialize, Serializer, de::Error as _};
+
+ use crate::{
+ AvroSchemaComponent, Schema,
+ schema::{Name, NamespaceRef, RecordField},
+ };
+
+ /// Returns `Schema::Array(T::get_schema_in_ctxt())`
+ pub fn get_schema_in_ctxt<T: AvroSchemaComponent>(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build()
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<const N: usize, S, T>(value: &[T; N], serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ T: Serialize,
+ {
+ value.as_slice().serialize(serializer)
+ }
+
+ pub fn deserialize<'de, const N: usize, D, T>(deserializer: D) -> Result<[T; N], D::Error>
+ where
+ D: Deserializer<'de>,
+ T: Deserialize<'de>,
+ {
+ let bytes = <Vec<T> as Deserialize>::deserialize(deserializer)?;
+ bytes.try_into().map_err(|v: Vec<T>| {
+ D::Error::custom(format!(
+ "Deserialized array has length {} which does not match array length of {N}",
+ v.len()
+ ))
+ })
+ }
+}
+
+/// (De)serialize an optional Rust array (`Option<[T; N]>`) as an Avro `Schema::Union([Schema::Null, Schema::Array])`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array`] for non-optional array values.
+///
+/// When used with different serialization formats, this is equivalent to [`serde_bytes`].
Review Comment:
Yes, sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3032068083
##########
avro/src/schema/union.rs:
##########
@@ -74,7 +81,116 @@ impl UnionSchema {
&self.schemas
}
- /// Returns true if the any of the variants of this `UnionSchema` is `Null`.
+ /// Get the variant at the given index.
+ pub fn get_variant(&self, index: usize) -> Result<&Schema, Error> {
+ self.schemas.get(index).ok_or_else(|| {
+ Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.schemas.len(),
+ }
+ .into()
+ })
+ }
+
+ /// Get the index of the provided [`SchemaKind`].
+ ///
+ /// The schema must not be a logical type, as only the base type are saved in the lookup index.
+ pub(crate) fn index_of_schema_kind(&self, kind: SchemaKind) -> Option<usize> {
+ self.variant_index.get(&kind).copied()
+ }
+
+ /// Get the index and schema for the provided name.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_named_schema<'s>(
+ &'s self,
+ name: &str,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s Schema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ if let Some(schema_name) = schema.name()
+ && schema_name.name() == name
Review Comment:
No, it is used for matching Rust type names to schemas which don't include the namespace. I've updated the documentation to make that more explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3032114125
##########
avro/src/reader/single_object.rs:
##########
@@ -315,11 +354,11 @@ mod tests {
fn avro_rs_164_generic_reader_alternate_header() -> TestResult {
let schema_uuid = Uuid::parse_str("b2f1cf00-0434-013e-439a-125eb8485a5f")?;
let header_builder = GlueSchemaUuidHeader::from_uuid(schema_uuid);
- let generic_reader = GenericSingleObjectReader::new_with_header_builder(
- TestSingleObjectReader::get_schema(),
- header_builder,
- )
- .expect("failed to build reader");
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .header(header_builder.build_header())
+ .build()
+ .expect("failed to build reader");
let data_to_read: Vec<u8> = vec![
3, 0, 178, 241, 207, 0, 4, 52, 1, 62, 67, 154, 18, 94, 184, 72, 90, 95,
];
Review Comment:
Adding a assert makes the test fail, I've reworked the test to actually test being able to deserialize a value wih a different header format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3032122502
##########
avro/src/reader/single_object.rs:
##########
@@ -315,19 +345,26 @@ mod tests {
fn avro_rs_164_generic_reader_alternate_header() -> TestResult {
let schema_uuid = Uuid::parse_str("b2f1cf00-0434-013e-439a-125eb8485a5f")?;
let header_builder = GlueSchemaUuidHeader::from_uuid(schema_uuid);
- let generic_reader = GenericSingleObjectReader::new_with_header_builder(
- TestSingleObjectReader::get_schema(),
- header_builder,
- )
- .expect("failed to build reader");
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .header(header_builder.build_header())
+ .build()
+ .expect("failed to build reader");
+ // First 18 bytes are the header, than it's a varint 0, double 0 and an empty list (0)
Review Comment:
```suggestion
// First 18 bytes are the header, then it's a varint 0, double 0 and an empty list (0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969298418
##########
avro/src/reader/block.rs:
##########
@@ -205,6 +215,43 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
+ ) -> AvroResult<Option<T>> {
+ if self.is_empty() {
+ self.read_block_next()?;
+ if self.is_empty() {
+ return Ok(None);
+ }
+ }
+
+ let mut block_bytes = &self.buf[self.buf_idx..];
+ let b_original = block_bytes.len();
+
+ let item = if read_schema.is_some() {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
Review Comment:
Follow-up, I've changed it to a panic and a comment. I've also documented on the builder that setting a reader schema is currently incompatible with the deserialisation interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4152863778
It is huge and I don't have time to review it right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969304859
##########
avro/src/schema/union.rs:
##########
@@ -74,7 +81,116 @@ impl UnionSchema {
&self.schemas
}
- /// Returns true if the any of the variants of this `UnionSchema` is `Null`.
+ /// Get the variant at the given index.
+ pub fn get_variant(&self, index: usize) -> Result<&Schema, Error> {
+ self.schemas.get(index).ok_or_else(|| {
+ Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.schemas.len(),
+ }
+ .into()
+ })
+ }
+
+ /// Get the index of the provided [`SchemaKind`].
+ ///
+ /// The schema must not be a logical type, as only the base type are saved in the lookup index.
+ pub(crate) fn index_of_schema_kind(&self, kind: SchemaKind) -> Option<usize> {
+ self.variant_index.get(&kind).copied()
+ }
+
+ /// Get the index and schema for the provided name.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_named_schema<'s>(
+ &'s self,
+ name: &str,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s Schema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ if let Some(schema_name) = schema.name()
+ && schema_name.name() == name
+ {
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ return Ok(Some((index, schema)));
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Fixed`] with the given size.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_fixed_of_size_n<'s>(
+ &'s self,
+ size: usize,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s FixedSchema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ match schema {
+ Schema::Fixed(fixed)
+ | Schema::Uuid(UuidSchema::Fixed(fixed))
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(fixed),
+ ..
+ })
+ | Schema::Duration(fixed)
+ if fixed.size == size =>
+ {
+ return Ok(Some((index, fixed)));
+ }
+ _ => {}
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Record`] with `n` fields.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_record_with_n_fields<'s>(
Review Comment:
It's used in the serializer for the `tuple` and `map` cases. We don't have a set name for a tuple, so we just use the first record with the same amount of fields.
For `serialize_map` it's used for records with flattened fields, because we don't get the name. This is an improvement on the previous serialize implementation that would just pick the first map/record/reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2966046326
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples`.
+//! - **tuple_variant** => See [Enums](##Enums)
+//! - **map** => [`Schema::Map`]
+//! - **Note:** the key type of the map will be (de)serialized as a [`Schema::String`]
+//! - **struct** => [`Schema::Record`]
+//! - **struct_variant** => See [Enums](##Enums)
+//!
+//! ## Enums
+//!
+//! ### Externally tagged
+//! This is the default enum representation for Serde. It can be mapped in three ways to the Avro data model.
+//! For all three options it is important that the enum index matches the Avro index.
+//! - As a [`Schema::Enum`], this is only possible for enums with only unit variants.
+//! - As a [`Schema::Union`] with a [`Schema::Record`] for every variant:
+//! - **unit_variant** => [`Schema::Record`] named as the variant and with no fields.
+//! - **newtype_variant** => [`Schema::Record`] named as the variant and with one field.
+//! The schema must have the attribute `org.apache.avro.rust.union_of_records` with the value set to `true`.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//! - As a [`Schema::Union`] without the wrapper [`Schema::Record`], all schemas must be unique:
+//! - **unit_variant** => [`Schema::Null`].
+//! - **newtype_variant** => The schema of the inner type.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//!
+//! ### Internally tagged
+//! This enum representation is used by Serde if the attribute `#[serde(tag = "...")]` is used.
+//! It maps to a [`Schema::Record`]. There must be at least one field that is named as the value of the
+//! `tag` attribute. If a field is not used by all variants it must have a `default` set.
+//!
+//! ### Adjacently tagged
+//! This enum representation is used by Serde if the attributes `#[serde(tag = "...", content = "...")]` are used.
+//! It maps to a [`Schema::Record`] with two fields. One field must be named as the value of the `tag`
+//! attribute and use the [`Schema::Enum`] schema. The other field must be named as the value of the
+//! `content` tag and use the [`Schema::Union`] schema.
+//!
+//! ### Untagged
+//! This enum representation is ued by Serde if the attribute `#[serde(untagged)]` is used. It maps
Review Comment:
```suggestion
//! This enum representation is used by Serde if the attribute `#[serde(untagged)]` is used. It maps
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2965456822
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
Review Comment:
this seems to point to the current module/document
```suggestion
//! see [the documentation for the reverse](super::avro_data_model_to_serde).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2974276596
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{DecimalSchema, InnerDecimalSchema, SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (
+ Value::String(_),
+ Schema::Bytes
+ | Schema::String
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ ..
+ })
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::String),
+ ) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.as_f64().is_some() => true,
+ (
+ Value::String(s),
+ Schema::Fixed(fixed)
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(fixed),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(fixed))
+ | Schema::Duration(fixed),
+ ) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.as_f64().is_some() => {
+ serializer.serialize_f32(n.as_f64().unwrap() as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.as_f64().is_some() => {
+ serializer.serialize_f64(n.as_f64().unwrap())
+ }
+ (
+ Value::String(s),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_))
+ | Schema::BigDecimal
+ | Schema::Decimal(_)
+ | Schema::Duration(_)
+ | Schema::Date,
Review Comment:
Why Date is here ?
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](#enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](#enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema of the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples`.
Review Comment:
```suggestion
//! is different from normal tuples.
```
##########
avro/src/serde/deser_schema/tuple.rs:
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, SeqAccess};
+
+use crate::{
+ Error, Schema,
+ schema::RecordSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer},
+};
+
+pub struct OneTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ field_read: bool,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> OneTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema {
+ config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ field_read: false,
+ })
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for OneTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.field_read {
+ Ok(None)
+ } else {
+ self.field_read = true;
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)
+ .map(Some)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ Some(1 - usize::from(self.field_read))
+ }
+}
+
+pub struct ManyTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> ManyTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for ManyTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.current_field < self.schema.fields.len() {
+ let schema = &self.schema.fields[self.current_field].schema;
+ self.current_field += 1;
Review Comment:
Let's move the incrementation after `seed.deserialize()`, so that it is not incremented if deserialize fails for any reason
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1817 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod block;
+mod enums;
+mod identifier;
+mod record;
+mod tuple;
+
+use block::BlockDeserializer;
+use enums::PlainEnumDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let variant = schema.get_variant(index)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u128(visitor),
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_f32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Float => visitor.visit_f32(f32::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f32(visitor),
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
+ }
+
+ fn deserialize_f64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Double => visitor.visit_f64(f64::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f64(visitor),
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
+ }
+
+ fn deserialize_char<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // A char cannot be deserialized using Schema::Uuid(UuidSchema::String) as that is at least
+ // 32 characters.
+ Schema::String => {
+ let string = self.read_string()?;
+ let mut chars = string.chars();
+ let char = chars
+ .next()
+ .ok_or_else(|| self.error("char", "String is empty"))?;
+ // We can't just check the string length, as that is in bytes not characters
+ if chars.next().is_some() {
+ Err(self.error(
+ "char",
+ format!(r#"Read more than one character: "{string}""#),
+ ))
+ } else {
+ visitor.visit_char(char)
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_char(visitor),
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_str<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_string(visitor)
+ }
+
+ fn deserialize_string<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ visitor.visit_string(self.read_string()?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_string(visitor),
+ _ => Err(self.error("string", "Expected Schema::String | Schema::Uuid(String)")),
+ }
+ }
+
+ fn deserialize_bytes<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_byte_buf(visitor)
+ }
+
+ fn deserialize_byte_buf<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ visitor.visit_byte_buf(self.read_bytes_with_len()?)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), ..}) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ visitor.visit_byte_buf(self.read_bytes(fixed.size)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_byte_buf(visitor),
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Fixed | Bytes) | Schema::Duration")),
+ }
+ }
+
+ fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && union.is_nullable()
+ {
+ let index = zag_i32(self.reader)?;
+ let schema = union.get_variant(index as usize)?;
Review Comment:
```suggestion
let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
let schema = union.get_variant(index)?;
```
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1817 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod block;
+mod enums;
+mod identifier;
+mod record;
+mod tuple;
+
+use block::BlockDeserializer;
+use enums::PlainEnumDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let variant = schema.get_variant(index)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u128(visitor),
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_f32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Float => visitor.visit_f32(f32::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f32(visitor),
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
+ }
+
+ fn deserialize_f64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Double => visitor.visit_f64(f64::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f64(visitor),
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
+ }
+
+ fn deserialize_char<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // A char cannot be deserialized using Schema::Uuid(UuidSchema::String) as that is at least
+ // 32 characters.
+ Schema::String => {
+ let string = self.read_string()?;
+ let mut chars = string.chars();
+ let char = chars
+ .next()
+ .ok_or_else(|| self.error("char", "String is empty"))?;
+ // We can't just check the string length, as that is in bytes not characters
+ if chars.next().is_some() {
+ Err(self.error(
+ "char",
+ format!(r#"Read more than one character: "{string}""#),
+ ))
+ } else {
+ visitor.visit_char(char)
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_char(visitor),
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_str<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_string(visitor)
+ }
+
+ fn deserialize_string<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ visitor.visit_string(self.read_string()?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_string(visitor),
+ _ => Err(self.error("string", "Expected Schema::String | Schema::Uuid(String)")),
+ }
+ }
+
+ fn deserialize_bytes<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_byte_buf(visitor)
+ }
+
+ fn deserialize_byte_buf<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ visitor.visit_byte_buf(self.read_bytes_with_len()?)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), ..}) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ visitor.visit_byte_buf(self.read_bytes(fixed.size)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_byte_buf(visitor),
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Fixed | Bytes) | Schema::Duration")),
+ }
+ }
+
+ fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && union.is_nullable()
+ {
+ let index = zag_i32(self.reader)?;
+ let schema = union.get_variant(index as usize)?;
+ if let Schema::Null = schema {
+ visitor.visit_none()
+ } else {
+ visitor.visit_some(self.with_different_schema(schema)?)
+ }
+ } else {
+ Err(self.error("option", "Expected Schema::Union([Schema::Null, _])"))
+ }
+ }
+
+ fn deserialize_unit<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => visitor.visit_unit(),
+ Schema::Union(union) => self.with_union(union)?.deserialize_unit(visitor),
+ _ => Err(self.error("unit", "Expected Schema::Null")),
+ }
+ }
+
+ fn deserialize_unit_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ visitor.visit_unit()
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_unit_struct(name, visitor),
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 0)"),
+ )),
+ }
+ }
+
+ fn deserialize_newtype_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == 1 && record.name.name() == name => {
+ visitor.visit_newtype_struct(self.with_different_schema(&record.fields[0].schema)?)
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_newtype_struct(name, visitor),
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 1)"),
+ )),
+ }
+ }
+
+ fn deserialize_seq<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Array(array) => {
+ visitor.visit_seq(BlockDeserializer::array(self.reader, array, self.config)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_seq(visitor),
+ _ => Err(self.error("seq", "Expected Schema::Array")),
+ }
+ }
+
+ fn deserialize_tuple<V>(self, len: usize, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // `len == 0` is not possible for derived Deserialize implementations but users might use it.
+ // The derived Deserialize implementations use `deserialize_unit` instead
+ Schema::Null if len == 0 => visitor.visit_unit(),
+ schema if len == 1 => {
+ visitor.visit_seq(OneTupleDeserializer::new(self.reader, schema, self.config)?)
+ }
+ Schema::Record(record) if record.fields.len() == len => {
+ visitor.visit_seq(ManyTupleDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_tuple(len, visitor),
+ _ if len == 0 => Err(self.error("tuple", "Expected Schema::Null for unit tuple")),
+ _ => Err(self.error(
+ "tuple",
+ format!("Expected Schema::Record(fields.len() == {len}) for {len}-tuple"),
+ )),
+ }
+ }
+
+ fn deserialize_tuple_struct<V>(
+ self,
+ name: &'static str,
+ len: usize,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.name.name() == name && record.fields.len() == len => {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
Review Comment:
Shouldn't this be `visit_seq()` ?
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1817 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod block;
+mod enums;
+mod identifier;
+mod record;
+mod tuple;
+
+use block::BlockDeserializer;
+use enums::PlainEnumDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let variant = schema.get_variant(index)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u128(visitor),
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_f32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Float => visitor.visit_f32(f32::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f32(visitor),
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
+ }
+
+ fn deserialize_f64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Double => visitor.visit_f64(f64::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f64(visitor),
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
+ }
+
+ fn deserialize_char<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // A char cannot be deserialized using Schema::Uuid(UuidSchema::String) as that is at least
+ // 32 characters.
+ Schema::String => {
+ let string = self.read_string()?;
+ let mut chars = string.chars();
+ let char = chars
+ .next()
+ .ok_or_else(|| self.error("char", "String is empty"))?;
+ // We can't just check the string length, as that is in bytes not characters
+ if chars.next().is_some() {
+ Err(self.error(
+ "char",
+ format!(r#"Read more than one character: "{string}""#),
+ ))
+ } else {
+ visitor.visit_char(char)
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_char(visitor),
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_str<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_string(visitor)
+ }
+
+ fn deserialize_string<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ visitor.visit_string(self.read_string()?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_string(visitor),
+ _ => Err(self.error("string", "Expected Schema::String | Schema::Uuid(String)")),
+ }
+ }
+
+ fn deserialize_bytes<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_byte_buf(visitor)
+ }
+
+ fn deserialize_byte_buf<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ visitor.visit_byte_buf(self.read_bytes_with_len()?)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), ..}) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ visitor.visit_byte_buf(self.read_bytes(fixed.size)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_byte_buf(visitor),
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Fixed | Bytes) | Schema::Duration")),
+ }
+ }
+
+ fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && union.is_nullable()
+ {
+ let index = zag_i32(self.reader)?;
+ let schema = union.get_variant(index as usize)?;
+ if let Schema::Null = schema {
+ visitor.visit_none()
+ } else {
+ visitor.visit_some(self.with_different_schema(schema)?)
+ }
+ } else {
+ Err(self.error("option", "Expected Schema::Union([Schema::Null, _])"))
+ }
+ }
+
+ fn deserialize_unit<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => visitor.visit_unit(),
+ Schema::Union(union) => self.with_union(union)?.deserialize_unit(visitor),
+ _ => Err(self.error("unit", "Expected Schema::Null")),
+ }
+ }
+
+ fn deserialize_unit_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ visitor.visit_unit()
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_unit_struct(name, visitor),
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 0)"),
+ )),
+ }
+ }
+
+ fn deserialize_newtype_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == 1 && record.name.name() == name => {
+ visitor.visit_newtype_struct(self.with_different_schema(&record.fields[0].schema)?)
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_newtype_struct(name, visitor),
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 1)"),
+ )),
+ }
+ }
+
+ fn deserialize_seq<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Array(array) => {
+ visitor.visit_seq(BlockDeserializer::array(self.reader, array, self.config)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_seq(visitor),
+ _ => Err(self.error("seq", "Expected Schema::Array")),
+ }
+ }
+
+ fn deserialize_tuple<V>(self, len: usize, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // `len == 0` is not possible for derived Deserialize implementations but users might use it.
+ // The derived Deserialize implementations use `deserialize_unit` instead
+ Schema::Null if len == 0 => visitor.visit_unit(),
+ schema if len == 1 => {
+ visitor.visit_seq(OneTupleDeserializer::new(self.reader, schema, self.config)?)
+ }
+ Schema::Record(record) if record.fields.len() == len => {
+ visitor.visit_seq(ManyTupleDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_tuple(len, visitor),
+ _ if len == 0 => Err(self.error("tuple", "Expected Schema::Null for unit tuple")),
+ _ => Err(self.error(
+ "tuple",
+ format!("Expected Schema::Record(fields.len() == {len}) for {len}-tuple"),
+ )),
+ }
+ }
+
+ fn deserialize_tuple_struct<V>(
+ self,
+ name: &'static str,
+ len: usize,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.name.name() == name && record.fields.len() == len => {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_tuple_struct(name, len, visitor),
+ _ => Err(self.error(
+ "tuple struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == {len})"),
+ )),
+ }
+ }
+
+ fn deserialize_map<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Map(map) => {
+ visitor.visit_map(BlockDeserializer::map(self.reader, map, self.config)?)
+ }
+ Schema::Record(record) => {
+ // Needed for flattened structs which are (de)serialized as maps
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_map(visitor),
+ _ => Err(self.error("map", "Expected Schema::Map")),
+ }
+ }
+
+ fn deserialize_struct<V>(
+ self,
+ name: &'static str,
+ fields: &'static [&'static str],
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // Checking that the amount of fields match does not work because of `skip_deserializing`
+ Schema::Record(record)
+ if record.name.name() == name || name.as_ptr() == DESERIALIZE_ANY.as_ptr() =>
+ {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_struct(name, fields, visitor),
+ _ => Err(self.error("struct", format!("Expected Schema::Record(name: {name})"))),
+ }
+ }
+
+ fn deserialize_enum<V>(
+ self,
+ _name: &'static str,
+ // This also includes aliases, so can't be used to check the amount of symbols
+ _variants: &'static [&'static str],
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Enum(schema) => {
+ visitor.visit_enum(PlainEnumDeserializer::new(self.reader, schema))
+ }
+ Schema::Union(union) => {
+ visitor.visit_enum(UnionEnumDeserializer::new(self.reader, union, self.config))
+ }
+ _ => Err(self.error("enum", "Expected Schema::Enum | Schema::Union")),
+ }
+ }
+
+ fn deserialize_identifier<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String => self.deserialize_string(visitor),
+ _ => Err(self.error("identifier", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_ignored_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ // TODO: Add `Seek` bound and skip over ignored data
+ self.deserialize_any(visitor)
+ }
+
+ fn is_human_readable(&self) -> bool {
+ self.config.human_readable
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::fmt::Debug;
+
+ use apache_avro_test_helper::TestResult;
+ use bigdecimal::BigDecimal;
+ use num_bigint::BigInt;
+ use pretty_assertions::assert_eq;
+ use serde::{Deserialize, Serialize, de::DeserializeOwned};
+ use serde_bytes::ByteBuf;
+ use uuid::Uuid;
+
+ use super::*;
+ use crate::{
+ AvroResult, AvroSchema, Decimal, reader::datum::GenericDatumReader,
+ writer::datum::GenericDatumWriter,
+ };
+
+ #[track_caller]
+ fn assert_roundtrip<T>(value: T, schema: &Schema, schemata: Vec<&Schema>) -> AvroResult<()>
+ where
+ T: Serialize + DeserializeOwned + PartialEq + Debug + Clone,
+ {
+ let buf = GenericDatumWriter::builder(schema)
+ .schemata(schemata.clone())?
+ .build()?
+ .write_ser_to_vec(&value)?;
+
+ let decoded_value: T = GenericDatumReader::builder(schema)
+ .writer_schemata(schemata)?
+ .build()?
+ .read_deser(&mut &buf[..])?;
+
+ assert_eq!(decoded_value, value);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3955_decode_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
+ pub enum SourceType {
+ Sozu,
+ Haproxy,
+ HaproxyTcp,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct AccessLog {
+ source: SourceType,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "AccessLog",
+ "namespace": "com.clevercloud.accesslogs.common.avro",
+ "type": "record",
+ "fields": [{
+ "name": "source",
+ "type": {
+ "type": "enum",
+ "name": "SourceType",
+ "items": "string",
+ "symbols": ["SOZU", "HAPROXY", "HAPROXY_TCP"]
+ }
+ }]
+ }"#,
+ )?;
+
+ let data = AccessLog {
+ source: SourceType::Sozu,
+ };
+
+ assert_roundtrip(data, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_decode_enum_invalid_data() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
+ pub enum SourceType {
+ Sozu,
+ Haproxy,
+ HaproxyTcp,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct AccessLog {
+ source: SourceType,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "AccessLog",
+ "namespace": "com.clevercloud.accesslogs.common.avro",
+ "type": "record",
+ "fields": [{
+ "name": "source",
+ "type": {
+ "type": "enum",
+ "name": "SourceType",
+ "items": "string",
+ "symbols": ["SOZU", "HAPROXY", "HAPROXY_TCP"]
+ }
+ }]
+ }"#,
+ )?;
+
+ // Contains index 3 (4th symbol)
+ let data_with_unknown_index = &[6u8];
+
+ let error = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser::<AccessLog>(&mut &data_with_unknown_index[..])
+ .unwrap_err();
+
+ assert_eq!(
+ error.to_string(),
+ "Enum symbol index out of bounds: got 3 but there are only 3 variants"
+ );
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_nested_struct() -> TestResult {
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
+ struct Test {
+ a: i64,
+ b: String,
+ c: Decimal,
+ }
+
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
+ struct TestInner {
+ a: Test,
+ b: i32,
+ }
+
+ let schemas = Schema::parse_list([
+ r#"{
+ "name": "Test",
+ "type": "record",
+ "fields": [
+ {
+ "name": "a",
+ "type": "long"
+ },
+ {
+ "name": "b",
+ "type": "string"
+ },
+ {
+ "name": "c",
+ "type": {
+ "type": "bytes",
+ "logicalType": "decimal",
+ "precision": 4,
+ "scale": 2
+ }
+ }
+ ]
+ }"#,
+ r#"{
+ "name": "TestInner",
+ "type": "record",
+ "fields": [
+ {
+ "name": "a",
+ "type": "Test"
+ },
+ {
+ "name": "b",
+ "type": "int"
+ }
+ ]
+ }"#,
+ ])?;
+
+ let test = Test {
+ a: 27,
+ b: "foo".to_string(),
+ c: Decimal::from(vec![1, 24]),
+ };
+
+ assert_roundtrip(test.clone(), &schemas[0], Vec::new())?;
+
+ let test_inner = TestInner { a: test, b: 35 };
+
+ assert_roundtrip(test_inner, &schemas[1], vec![&schemas[0]])?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_external_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ pub enum UnitExternalEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitExternalEnum {
+ a: UnitExternalEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "enum",
+ "name": "UnitExternalEnum",
+ "items": "string",
+ "symbols": ["Val1", "Val2"]
+ }
+ }]
+ }"#,
+ )?;
+
+ let alt_schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": []
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitExternalEnum {
+ a: UnitExternalEnum::Val1,
+ };
+ assert_roundtrip(value.clone(), &schema, Vec::new())?;
+ assert_roundtrip(value, &alt_schema, Vec::new())?;
+
+ let value = TestUnitExternalEnum {
+ a: UnitExternalEnum::Val2,
+ };
+ assert_roundtrip(value.clone(), &alt_schema, Vec::new())?;
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_internal_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(tag = "t")]
+ pub enum UnitInternalEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitInternalEnum {
+ a: UnitInternalEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitInternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "record",
+ "name": "UnitInternalEnum",
+ "fields": [{
+ "name": "t",
+ "type": "string"
+ }]
+ }
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitInternalEnum {
+ a: UnitInternalEnum::Val1,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ let value = TestUnitInternalEnum {
+ a: UnitInternalEnum::Val2,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_adjacent_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(tag = "t", content = "v")]
+ pub enum UnitAdjacentEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitAdjacentEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "record",
+ "name": "UnitAdjacentEnum",
+ "fields": [
+ {
+ "name": "t",
+ "type": {
+ "type": "enum",
+ "name": "t",
+ "symbols": ["Val1", "Val2"]
+ }
+ },
+ {
+ "name": "v",
+ "default": null,
+ "type": ["null"]
+ }
+ ]
+ }
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum::Val1,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ let value = TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum::Val2,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_untagged_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(untagged)]
+ pub enum UnitUntaggedEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitUntaggedEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": ["null"]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum::Val1,
+ };
+ assert_roundtrip(value1.clone(), &schema, Vec::new())?;
+
+ let value2 = TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum::Val2,
+ };
+ let buf = GenericDatumWriter::builder(&schema)
+ .build()?
+ .write_ser_to_vec(&value1)?;
+
+ let decoded_value: TestUnitUntaggedEnum = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser(&mut &buf[..])?;
+
+ // Val2 cannot troundtrip. All unit variants are serialized to the same null.
+ // This also doesn't roundtrip in serde_json.
+ assert_ne!(value2, decoded_value);
+ assert_eq!(decoded_value, value1);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_mixed_enum() -> TestResult {
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq)]
+ struct TestNullExternalEnum {
+ a: NullExternalEnum,
+ }
+
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq)]
+ enum NullExternalEnum {
+ Val1,
+ Val2(),
+ Val3(()),
+ Val4(u64),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestNullExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val3",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "null"
+ }]
+ },
+ {
+ "name": "Val4",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": {
+ "type": "fixed",
+ "name": "u64",
+ "size": 8
+ }
+ }]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let data = [
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val1,
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val2(),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val2(),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val3(()),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val4(123),
+ },
+ ];
+
+ for value in data {
+ assert_roundtrip(value, &schema, Vec::new())?;
+ }
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_single_value_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum SingleValueExternalEnum {
+ Double(f64),
+ String(String),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestSingleValueExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Double",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "double"
+ }]
+ },
+ {
+ "name": "String",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "string"
+ }]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let alt_schema = Schema::parse_str(
+ r#"{
+ "name": "TestSingleValueExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": ["double", "string"]
+ }]
+ }"#,
+ )?;
+
+ let double = TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum::Double(64.0),
+ };
+ assert_roundtrip(double.clone(), &schema, Vec::new())?;
+ assert_roundtrip(double, &alt_schema, Vec::new())?;
+
+ let string = TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum::String("test".to_string()),
+ };
+ assert_roundtrip(string.clone(), &schema, Vec::new())?;
+ assert_roundtrip(string, &alt_schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_struct_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestStructExternalEnum {
+ a: StructExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum StructExternalEnum {
+ Val1 { x: f32, y: f32 },
+ Val2 { x: f32, y: f32 },
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestStructExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "x",
+ "type": "float"
+ },
+ {
+ "name": "y",
+ "type": "float"
+ }
+ ]
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": [
+ {
+ "name": "x",
+ "type": "float"
+ },
+ {
+ "name": "y",
+ "type": "float"
+ }
+ ]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestStructExternalEnum {
+ a: StructExternalEnum::Val1 { x: 1.0, y: 2.0 },
+ };
+
+ assert_roundtrip(value1, &schema, Vec::new())?;
+
+ let value2 = TestStructExternalEnum {
+ a: StructExternalEnum::Val2 { x: 2.0, y: 1.0 },
+ };
+
+ assert_roundtrip(value2, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_struct_flatten() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct S1 {
+ f1: String,
+ #[serde(flatten)]
+ inner: S2,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct S2 {
+ f2: String,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "S1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "f1",
+ "type": "string"
+ },
+ {
+ "name": "f2",
+ "type": "string"
+ }
+ ]
+ }"#,
+ )?;
+
+ let value = S1 {
+ f1: "Hello".to_owned(),
+ inner: S2 {
+ f2: "World".to_owned(),
+ },
+ };
+
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_512_tuple_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestTupleExternalEnum {
+ a: TupleExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum TupleExternalEnum {
+ Val1(f32, f32),
+ Val2(f32, f32, f32),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestTupleExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "field_0",
+ "type": "float"
+ },
+ {
+ "name": "field_1",
+ "type": "float"
+ }
+ ]
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": [
+ {
+ "name": "field_0",
+ "type": "float"
+ },
+ {
+ "name": "field_1",
+ "type": "float"
+ },
+ {
+ "name": "field_2",
+ "type": "float"
+ }
+ ]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestTupleExternalEnum {
+ a: TupleExternalEnum::Val1(1.0, 2.0),
+ };
+
+ assert_roundtrip(value1, &schema, Vec::new())?;
+
+ let value2 = TestTupleExternalEnum {
+ a: TupleExternalEnum::Val1(2.0, 1.0),
Review Comment:
Val2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2972066848
##########
avro/src/schema/mod.rs:
##########
@@ -784,6 +785,69 @@ impl Schema {
}
Ok(())
}
+
+ /// Derive a name for this schema.
+ ///
+ /// The name is a valid schema name and will be unique if the named
+ /// schemas in this schema have unique names.
+ pub(crate) fn unique_normalized_name(&self) -> Cow<'static, str> {
+ match self {
+ Schema::Null => Cow::Borrowed("n"),
+ Schema::Boolean => Cow::Borrowed("B"),
+ Schema::Int => Cow::Borrowed("i"),
+ Schema::Long => Cow::Borrowed("l"),
+ Schema::Float => Cow::Borrowed("f"),
+ Schema::Double => Cow::Borrowed("d"),
+ Schema::Bytes => Cow::Borrowed("b"),
+ Schema::String => Cow::Borrowed("s"),
+ Schema::Array(array) => {
+ Cow::Owned(format!("a_{}", array.items.unique_normalized_name()))
+ }
+ Schema::Map(map) => Cow::Owned(format!("m_{}", map.types.unique_normalized_name())),
+ Schema::Union(union) => {
+ let mut name = format!("u{}", union.schemas.len());
+ for schema in &union.schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ Cow::Owned(name)
+ }
+ Schema::BigDecimal => Cow::Borrowed("bd"),
+ Schema::Date => Cow::Borrowed("D"),
+ Schema::TimeMillis => Cow::Borrowed("t"),
+ Schema::TimeMicros => Cow::Borrowed("tm"),
+ Schema::TimestampMillis => Cow::Borrowed("T"),
+ Schema::TimestampMicros => Cow::Borrowed("TM"),
+ Schema::TimestampNanos => Cow::Borrowed("TN"),
+ Schema::LocalTimestampMillis => Cow::Borrowed("L"),
+ Schema::LocalTimestampMicros => Cow::Borrowed("LM"),
+ Schema::LocalTimestampNanos => Cow::Borrowed("LN"),
+ Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ precision,
+ scale,
+ }) => Cow::Owned(format!("db_{precision}_{scale}")),
+ Schema::Uuid(UuidSchema::Bytes) => Cow::Borrowed("ub"),
+ Schema::Uuid(UuidSchema::String) => Cow::Borrowed("us"),
+ Schema::Record(RecordSchema { name, .. })
+ | Schema::Enum(EnumSchema { name, .. })
+ | Schema::Fixed(FixedSchema { name, .. })
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(FixedSchema { name, .. }),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(FixedSchema { name, .. }))
+ | Schema::Duration(FixedSchema { name, .. })
+ | Schema::Ref { name } => {
+ let name: String = name
+ .to_string()
+ .chars()
+ .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
Review Comment:
True, the only thing I can think of is hashing the names but that will make the names very ugly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969313698
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.is_f64() => {
+ let float = n.as_f64().unwrap();
+ serializer.serialize_f32(float as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.is_f64() => {
+ let double = n.as_f64().unwrap();
+ serializer.serialize_f64(double)
+ }
+ (
+ Value::String(s),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_))
+ | Schema::BigDecimal
+ | Schema::Decimal(_)
+ | Schema::Duration(_)
+ | Schema::Date,
+ ) => serializer.serialize_bytes(s.as_bytes()),
+ (Value::String(s), Schema::String | Schema::Uuid(UuidSchema::String)) => {
+ serializer.serialize_str(s)
+ }
+ (Value::String(s), Schema::Enum(enum_schema)) => {
+ let Some((variant_index, _)) = enum_schema
+ .symbols
+ .iter()
+ .enumerate()
+ .find(|(_i, symbol)| *symbol == s)
+ else {
+ return Err(S::Error::custom(format!(
+ "Could not find `{s}` in enum: {enum_schema:?}"
+ )));
+ };
+
+ serializer.serialize_unit_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ variant_index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ )
+ }
+ // This abuses the support for flattened fields, which are also serialized as a map.
+ (Value::Object(o), Schema::Record(record)) => serializer.collect_map(
+ o.iter()
+ .enumerate()
+ .map(|(i, (k, v))| (k, Self::new(v, &record.fields[i].schema))),
+ ),
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969309754
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969299749
##########
avro/src/reader/block.rs:
##########
@@ -205,6 +215,43 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
+ ) -> AvroResult<Option<T>> {
+ if self.is_empty() {
+ self.read_block_next()?;
+ if self.is_empty() {
+ return Ok(None);
+ }
+ }
+
+ let mut block_bytes = &self.buf[self.buf_idx..];
+ let b_original = block_bytes.len();
+
+ let item = if read_schema.is_some() {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
+ } else {
+ let config = Config {
+ names: &self.names_refs,
+ human_readable: self.human_readable,
+ };
+ T::deserialize(SchemaAwareDeserializer::new(
+ &mut block_bytes,
+ &self.writer_schema,
+ config,
+ )?)?
+ };
+
+ if b_original != 0 && b_original == block_bytes.len() {
+ // No bytes were read, return an error to avoid an infinite loop
+ return Err(Details::ReadBlock.into());
Review Comment:
I've changed the error description from `unable to read block` to `Did not read any bytes, block is corrupt`.
The only way that this can be triggered if the schema is `Schema::Null` and the block actually has data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4097259740
Forgot the doctests, I'll fix them after lunch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4183444518
Thank you so much for the time and effort you put into reviewing this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969301270
##########
avro/src/schema/mod.rs:
##########
@@ -784,6 +785,69 @@ impl Schema {
}
Ok(())
}
+
+ /// Derive a name for this schema.
+ ///
+ /// The name is a valid schema name and will be unique if the named
+ /// schemas in this schema have unique names.
+ pub(crate) fn unique_normalized_name(&self) -> Cow<'static, str> {
+ match self {
+ Schema::Null => Cow::Borrowed("n"),
+ Schema::Boolean => Cow::Borrowed("B"),
+ Schema::Int => Cow::Borrowed("i"),
+ Schema::Long => Cow::Borrowed("l"),
+ Schema::Float => Cow::Borrowed("f"),
+ Schema::Double => Cow::Borrowed("d"),
+ Schema::Bytes => Cow::Borrowed("b"),
+ Schema::String => Cow::Borrowed("s"),
+ Schema::Array(array) => {
+ Cow::Owned(format!("a_{}", array.items.unique_normalized_name()))
+ }
+ Schema::Map(map) => Cow::Owned(format!("m_{}", map.types.unique_normalized_name())),
+ Schema::Union(union) => {
+ let mut name = format!("u{}", union.schemas.len());
+ for schema in &union.schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ Cow::Owned(name)
+ }
+ Schema::BigDecimal => Cow::Borrowed("bd"),
+ Schema::Date => Cow::Borrowed("D"),
+ Schema::TimeMillis => Cow::Borrowed("t"),
+ Schema::TimeMicros => Cow::Borrowed("tm"),
+ Schema::TimestampMillis => Cow::Borrowed("T"),
+ Schema::TimestampMicros => Cow::Borrowed("TM"),
+ Schema::TimestampNanos => Cow::Borrowed("TN"),
+ Schema::LocalTimestampMillis => Cow::Borrowed("L"),
+ Schema::LocalTimestampMicros => Cow::Borrowed("LM"),
+ Schema::LocalTimestampNanos => Cow::Borrowed("LN"),
+ Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ precision,
+ scale,
+ }) => Cow::Owned(format!("db_{precision}_{scale}")),
+ Schema::Uuid(UuidSchema::Bytes) => Cow::Borrowed("ub"),
+ Schema::Uuid(UuidSchema::String) => Cow::Borrowed("us"),
+ Schema::Record(RecordSchema { name, .. })
+ | Schema::Enum(EnumSchema { name, .. })
+ | Schema::Fixed(FixedSchema { name, .. })
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(FixedSchema { name, .. }),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(FixedSchema { name, .. }))
+ | Schema::Duration(FixedSchema { name, .. })
+ | Schema::Ref { name } => {
+ let name: String = name
+ .to_string()
+ .chars()
+ .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
Review Comment:
Every `_` is now expended to two `_` and I've added some tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2971478896
##########
avro/src/serde/deser_schema/record.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, MapAccess};
+
+use crate::{
+ Error, Schema,
+ schema::RecordSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer, identifier::IdentifierDeserializer},
+};
+
+pub struct RecordDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> RecordDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> MapAccess<'de> for RecordDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_key_seed<K>(&mut self, seed: K) -> Result<Option<K::Value>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ {
+ if self.current_field >= self.schema.fields.len() {
+ // Finished reading this record
+ Ok(None)
+ } else {
+ seed.deserialize(IdentifierDeserializer::string(
+ &self.schema.fields[self.current_field].name,
+ ))
+ .map(Some)
+ }
+ }
+
+ fn next_value_seed<V>(&mut self, seed: V) -> Result<V::Value, Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let schema = &self.schema.fields[self.current_field].schema;
+ let value = if let Schema::Record(record) = schema
Review Comment:
What about `Schema::Ref` ?
##########
avro/src/schema/mod.rs:
##########
@@ -784,6 +785,69 @@ impl Schema {
}
Ok(())
}
+
+ /// Derive a name for this schema.
+ ///
+ /// The name is a valid schema name and will be unique if the named
+ /// schemas in this schema have unique names.
+ pub(crate) fn unique_normalized_name(&self) -> Cow<'static, str> {
+ match self {
+ Schema::Null => Cow::Borrowed("n"),
+ Schema::Boolean => Cow::Borrowed("B"),
+ Schema::Int => Cow::Borrowed("i"),
+ Schema::Long => Cow::Borrowed("l"),
+ Schema::Float => Cow::Borrowed("f"),
+ Schema::Double => Cow::Borrowed("d"),
+ Schema::Bytes => Cow::Borrowed("b"),
+ Schema::String => Cow::Borrowed("s"),
+ Schema::Array(array) => {
+ Cow::Owned(format!("a_{}", array.items.unique_normalized_name()))
+ }
+ Schema::Map(map) => Cow::Owned(format!("m_{}", map.types.unique_normalized_name())),
+ Schema::Union(union) => {
+ let mut name = format!("u{}", union.schemas.len());
+ for schema in &union.schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ Cow::Owned(name)
+ }
+ Schema::BigDecimal => Cow::Borrowed("bd"),
+ Schema::Date => Cow::Borrowed("D"),
+ Schema::TimeMillis => Cow::Borrowed("t"),
+ Schema::TimeMicros => Cow::Borrowed("tm"),
+ Schema::TimestampMillis => Cow::Borrowed("T"),
+ Schema::TimestampMicros => Cow::Borrowed("TM"),
+ Schema::TimestampNanos => Cow::Borrowed("TN"),
+ Schema::LocalTimestampMillis => Cow::Borrowed("L"),
+ Schema::LocalTimestampMicros => Cow::Borrowed("LM"),
+ Schema::LocalTimestampNanos => Cow::Borrowed("LN"),
+ Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ precision,
+ scale,
+ }) => Cow::Owned(format!("db_{precision}_{scale}")),
+ Schema::Uuid(UuidSchema::Bytes) => Cow::Borrowed("ub"),
+ Schema::Uuid(UuidSchema::String) => Cow::Borrowed("us"),
+ Schema::Record(RecordSchema { name, .. })
+ | Schema::Enum(EnumSchema { name, .. })
+ | Schema::Fixed(FixedSchema { name, .. })
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(FixedSchema { name, .. }),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(FixedSchema { name, .. }))
+ | Schema::Duration(FixedSchema { name, .. })
+ | Schema::Ref { name } => {
+ let name: String = name
+ .to_string()
+ .chars()
+ .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
Review Comment:
Both `a._b` and `a_.b` would still produce `a___b`, but I also don't see a way to avoid all duplicates.
And I doubt someone will use such names.
##########
avro/src/serde/deser_schema/array.rs:
##########
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, SeqAccess};
+
+use super::{Config, SchemaAwareDeserializer};
+use crate::{
+ Error, Schema,
+ schema::ArraySchema,
+ util::{zag_i32, zag_i64},
+};
+
+/// Deserialize sequences from an Avro array.
+pub struct ArrayDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ /// Track where we are in reading the array
+ remaining: Option<u32>,
Review Comment:
https://avro.apache.org/docs/1.12.0/specification/#complex-types-1 says:
```
Arrays are encoded as a series of blocks. Each block consists of a long count value, followed by that many array items.
```
"**long** count".
Also see https://github.com/apache/avro-rs/blob/6e6da17ade56087ed10b2064ef6817d90196f6a8/avro/src/encode.rs#L238
```suggestion
remaining: Option<u64>,
```
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol must match the index of the variant
+//! - `arrays`: [`Vec`] (or any type that uses [`Serialize::serialize_seq`](serde::Serialize), [`Deserialize::deserialize_seq`](serde::Deserialize))
+//! - `[T; N]` is (de)serialized as a tuple by Serde, to (de)serialize them as an `array` use [`apache_avro::serde::array`]
+//! - `maps`: [`HashMap<String, _>`](std::collections::HashMap) (or any type that uses [`Serialize::serialize_map`](serde::Serialize), [`Deserialize::deserialize_map`](serde::Deserialize))
+//! - `unions`: An enum with a variant for each union variant
+//! - The index of the union variant must match the enum variant
+//! - A `null` can be a unit variant or a newtype variant with a unit type
+//! - All other variants must be newtype variants, struct variants, or tuple variants.
+//! - `fixed`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
Review Comment:
```suggestion
//! - It is required to use [`apache_avro::serde::fixed`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
```
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
Review Comment:
```suggestion
Err(self.error("f32", "Expected Schema::Float in variants"))
```
##########
avro/src/serde/deser_schema/record.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, MapAccess};
+
+use crate::{
+ Error, Schema,
+ schema::RecordSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer, identifier::IdentifierDeserializer},
+};
+
+pub struct RecordDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> RecordDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> MapAccess<'de> for RecordDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_key_seed<K>(&mut self, seed: K) -> Result<Option<K::Value>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ {
+ if self.current_field >= self.schema.fields.len() {
+ // Finished reading this record
+ Ok(None)
+ } else {
+ seed.deserialize(IdentifierDeserializer::string(
+ &self.schema.fields[self.current_field].name,
+ ))
+ .map(Some)
+ }
+ }
+
+ fn next_value_seed<V>(&mut self, seed: V) -> Result<V::Value, Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let schema = &self.schema.fields[self.current_field].schema;
Review Comment:
Let's check whether `self.current_field` is not out of bounds.
##########
avro/src/serde/deser_schema/map.rs:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Read};
+
+use serde::de::{DeserializeSeed, MapAccess};
+
+use crate::{
+ Error, Schema,
+ schema::MapSchema,
+ serde::deser_schema::{Config, SchemaAwareDeserializer},
+ util::{zag_i32, zag_i64},
+};
+
+pub struct MapDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ remaining: Option<u32>,
Review Comment:
```suggestion
remaining: Option<u64>,
```
https://avro.apache.org/docs/1.12.0/specification/#complex-types-1 says:
```
Maps are encoded as a series of blocks. Each block consists of a long count value, ...
```
Note: **long**
Also see https://github.com/apache/avro-rs/blob/6e6da17ade56087ed10b2064ef6817d90196f6a8/avro/src/encode.rs#L264
##########
avro/src/serde/with.rs:
##########
@@ -503,11 +500,326 @@ pub mod slice_opt {
}
}
+/// (De)serialize [`BigDecimal`] as a [`Schema::BigDecimal`] instead of a [`Schema::String`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal_opt`] for optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal")]
+/// decimal: BigDecimal,
+/// }
+/// ```
+///
+/// [`BigDecimal`]: ::bigdecimal::BigDecimal
+/// [`Schema::BigDecimal`]: crate::Schema::BigDecimal
+/// [`Schema::String`]: crate::Schema::String
+/// [`apache_avro::serde::bigdecimal_opt`]: bigdecimal_opt
+pub mod bigdecimal {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField},
+ serde::with::BytesType,
+ };
+
+ /// Returns [`Schema::BigDecimal`]
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::BigDecimal
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &BigDecimal, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&decimal_bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<BigDecimal, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ // We don't use &'de [u8] here as the deserializer doesn't support that
+ let bytes: Vec<u8> = serde_bytes::deserialize(deserializer)?;
+
+ deserialize_big_decimal(&bytes).map_err(D::Error::custom)
+ }
+}
+
+/// (De)serialize [`Option<BigDecimal>`] as a `Schema::Union(Schema::Null, Schema::BigDecimal)` instead of a `Schema::Union(Schema::Null, Schema::String)`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal`] for non-optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::{AvroSchema, BigDecimal};
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal_opt")]
+/// decimal: Option<BigDecimal>,
+/// }
+/// ```
+///
+/// [`Option<BigDecimal>`]: ::bigdecimal::BigDecimal
+/// [`apache_avro::serde::bigdecimal`]: bigdecimal
+pub mod bigdecimal_opt {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField, UnionSchema},
+ serde::with::BytesType,
+ };
+
+ /// Returns `Schema::Union(Schema::Null, Schema::BigDecimal)`
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::Union(
+ UnionSchema::new(vec![Schema::Null, Schema::BigDecimal])
+ .expect("This is a valid union"),
+ )
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &Option<BigDecimal>, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ if let Some(decimal) = decimal {
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&Some(decimal_bytes), serializer)
+ } else {
+ serde_bytes::serialize(&None::<Vec<u8>>, serializer)
+ }
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<BigDecimal>, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ let bytes: Option<Vec<u8>> = serde_bytes::deserialize(deserializer)?;
+ if let Some(bytes) = bytes {
+ deserialize_big_decimal(&bytes)
+ .map(Some)
+ .map_err(D::Error::custom)
+ } else {
+ Ok(None)
+ }
+ }
+}
+
+/// (De)serialize an Rust array (`[T; N]`) as an Avro [`Schema::Array`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array_opt`] for optional array values.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithArray {
+/// #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<i32>)]
+/// #[serde(with = "apache_avro::serde::array")]
+/// array: [i32; 10],
+/// }
+/// ```
+///
+/// [`apache_avro::serde::array_opt`]: array_opt
+/// [`Schema::Array`]: crate::schema::Schema::Array
+pub mod array {
+ use std::collections::HashSet;
+
+ use serde::{
+ Deserialize, Deserializer, Serialize, Serializer,
+ de::{DeserializeOwned, Error as _},
+ };
+
+ use crate::{
+ AvroSchemaComponent, Schema,
+ schema::{Name, NamespaceRef, RecordField},
+ };
+
+ /// Returns `Schema::Array(T::get_schema_in_ctxt())`
+ pub fn get_schema_in_ctxt<T: AvroSchemaComponent>(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build()
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<const N: usize, S, T>(value: &[T; N], serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ T: Serialize,
+ {
+ value.as_slice().serialize(serializer)
+ }
+
+ pub fn deserialize<'de, const N: usize, D, T>(deserializer: D) -> Result<[T; N], D::Error>
+ where
+ D: Deserializer<'de>,
+ T: DeserializeOwned,
Review Comment:
```suggestion
T: Deserialize<'de>,
```
Does this work too ?
array_opt below is OK with it.
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol must match the index of the variant
+//! - `arrays`: [`Vec`] (or any type that uses [`Serialize::serialize_seq`](serde::Serialize), [`Deserialize::deserialize_seq`](serde::Deserialize))
+//! - `[T; N]` is (de)serialized as a tuple by Serde, to (de)serialize them as an `array` use [`apache_avro::serde::array`]
+//! - `maps`: [`HashMap<String, _>`](std::collections::HashMap) (or any type that uses [`Serialize::serialize_map`](serde::Serialize), [`Deserialize::deserialize_map`](serde::Deserialize))
+//! - `unions`: An enum with a variant for each union variant
+//! - The index of the union variant must match the enum variant
+//! - A `null` can be a unit variant or a newtype variant with a unit type
+//! - All other variants must be newtype variants, struct variants, or tuple variants.
+//! - `fixed`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
Review Comment:
```suggestion
//! - `fixed`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serializer::serialize_bytes`](serde::Serializer), [`Deserializer::deserialize_bytes`](serde::Deserializer), [`Deserializer::deserialize_byte_buf`](serde::Deserializer))
```
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
+ }
+ }
+
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Double) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f64", "Expected Schema::Double in variants"))?
+ }
+ }
+
+ fn serialize_char(mut self, v: char) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.to_string().as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("char", "Expected Schema::String in variants"))?
+ }
+ }
+
+ fn serialize_str(mut self, v: &str) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("str", "Expected Schema::String in variants"))?
Review Comment:
```suggestion
Err(self.error("str", "Expected Schema::String in variants"))
```
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
Review Comment:
```suggestion
//! - `string`: [`String`] (or any type that uses [`Serializer::serialize_str`](serde::Serializer), [`Deserializer::deserialize_str`](serde::Deserializer), [`Deserializer::deserialize_string`](serde::Deserializer))
```
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
+ }
+ }
+
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Double) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f64", "Expected Schema::Double in variants"))?
+ }
+ }
+
+ fn serialize_char(mut self, v: char) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.to_string().as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("char", "Expected Schema::String in variants"))?
+ }
+ }
+
+ fn serialize_str(mut self, v: &str) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("str", "Expected Schema::String in variants"))?
+ }
+ }
+
+ fn serialize_bytes(mut self, v: &[u8]) -> Result<Self::Ok, Self::Error> {
+ let (index, with_len) = match SER_BYTES_TYPE.get() {
+ BytesType::Bytes => {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Bytes) {
+ (index, true)
+ } else {
+ return Err(self.error("bytes", "Expected Schema::Bytes in variants"));
+ }
+ }
+ BytesType::Fixed => {
+ if let Some((index, _)) = self
+ .union
+ .find_fixed_of_size_n(v.len(), self.config.names)?
+ {
+ (index, false)
+ } else {
+ return Err(self.error(
+ "bytes",
+ format!("Expected Schema::Fixed(size: {}) in variants", v.len()),
+ ));
+ }
+ }
+ BytesType::Unset => {
+ let bytes_index = self.union.index_of_schema_kind(SchemaKind::Bytes);
+ let fixed_index = self
+ .union
+ .find_fixed_of_size_n(v.len(), self.config.names)?;
+ // Find the first variant that matches the bytes or fixed
+ match (bytes_index, fixed_index) {
+ (Some(bytes_index), Some((fixed_index, _))) => {
+ (bytes_index.min(fixed_index), bytes_index < fixed_index)
+ }
+ (Some(bytes_index), None) => (bytes_index, true),
+ (None, Some((fixed_index, _))) => (fixed_index, false),
+ (None, None) => {
+ return Err(self.error(
+ "bytes",
+ format!(
+ "Expected Schema::Bytes | Schema::Fixed(size: {}) in variants",
+ v.len()
+ ),
+ ));
+ }
+ }
+ }
+ };
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ if with_len {
+ bytes_written += self.write_bytes_with_len(v)?;
+ } else {
+ bytes_written += self.write_bytes(v)?;
+ }
+ Ok(bytes_written)
+ }
+
+ fn serialize_none(self) -> Result<Self::Ok, Self::Error> {
+ Err(self.error("none", "Nested unions are not supported"))
+ }
+
+ fn serialize_some<T>(self, _: &T) -> Result<Self::Ok, Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ Err(self.error("some", "Nested unions are not supported"))
+ }
+
+ fn serialize_unit(self) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Null) {
+ zig_i32(index as i32, &mut *self.writer)
+ } else {
+ Err(self.error("unit", "Expected Schema::Null in variants"))
+ }
+ }
+
+ fn serialize_unit_struct(self, name: &'static str) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Record(record))) if record.fields.is_empty() => {
+ zig_i32(index as i32, &mut *self.writer)
+ }
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields: []) in variants"),
+ )),
+ }
+ }
+
+ fn serialize_unit_variant(
+ self,
+ name: &'static str,
+ variant_index: u32,
+ variant: &'static str,
+ ) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Enum(schema))) if schema.symbols[variant_index as usize] == variant => {
Review Comment:
```suggestion
Some((index, Schema::Enum(schema))) if schema.symbols.get(variant_index as usize) == Some(&variant.to_string()) => {
```
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema of the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples`.
+//! - **tuple_variant** => See [Enums](##Enums)
Review Comment:
```suggestion
//! - **tuple_variant** => See [Enums](#enums)
```
##########
avro/src/serde/ser_schema/block.rs:
##########
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeSeq},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ encode::encode_int,
+ error::Details,
+ schema::{ArraySchema, MapSchema},
+ util::zig_i32,
+};
+
+#[expect(
+ private_interfaces,
+ reason = "Direct and Buffered should not be used directly"
+)]
+pub enum BlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Direct(DirectBlockSerializer<'s, 'w, W, S>),
+ Buffered(BufferedBlockSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BlockSerializer<'s, 'w, W, S> {
+ pub fn array(
+ writer: &'w mut W,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let bytes_written = bytes_written.unwrap_or(0);
+ if let Some(len) = len
+ && config.target_block_size.is_none()
+ {
+ Ok(Self::Direct(DirectBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ } else {
+ let target_block_size = config.target_block_size.unwrap_or(1024);
+ Ok(Self::Buffered(BufferedBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ target_block_size,
+ bytes_written,
+ )))
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_element(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_key(key),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_value(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+struct DirectBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> DirectBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: usize,
+ mut bytes_written: usize,
+ ) -> Result<Self, Error> {
+ if len != 0 {
+ // .end() always writes the zero block, so we only write the size for arrays
+ // that have at least one element
+ bytes_written += zig_i32(len as i32, &mut *writer)?;
+ }
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ bytes_written,
+ })
+ }
+
+ fn end(self) -> Result<usize, Error> {
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+struct BufferedBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ buffer: Vec<u8>,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+ items_in_buffer: usize,
+ target_block_size: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BufferedBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ target_block_size: usize,
+ bytes_written: usize,
+ ) -> Self {
+ Self {
+ writer,
+ buffer: Vec::with_capacity(target_block_size),
+ schema,
+ config,
+ bytes_written,
+ items_in_buffer: 0,
+ target_block_size,
+ }
+ }
+
+ /// Write a block including the items and size header.
+ fn write_block(&mut self) -> Result<(), Error> {
+ // Write the header, the negative item count indicates that the next value is the size of the
+ // block in bytes
+ self.bytes_written += encode_int(0 - (self.items_in_buffer as i32), &mut *self.writer)?;
Review Comment:
Why `items_in_buffer` is `usize` if we assume that it won't be bigger than i32 ?
##########
avro/src/serde/derive.rs:
##########
@@ -630,27 +612,35 @@ where
}
impl AvroSchemaComponent for core::time::Duration {
- /// The schema is [`Schema::Duration`] with the name `duration`.
+ /// The schema is [`Schema::Record`] with the name `Duration`.
///
- /// This is a lossy conversion as this Avro type does not store the amount of nanoseconds.
+ /// It has two fields:
+ /// - `secs` with the schema `Schema::Fixed(name: "u64", size: 8)`
+ /// - `nanos` with the schema `Schema::Long`
fn get_schema_in_ctxt(
named_schemas: &mut HashSet<Name>,
enclosing_namespace: NamespaceRef,
) -> Schema {
- let name = Name::new_with_enclosing_namespace("duration", enclosing_namespace)
+ let name = Name::new_with_enclosing_namespace("Duration", enclosing_namespace)
.expect("Name is valid");
if named_schemas.contains(&name) {
Schema::Ref { name }
} else {
- let schema = Schema::Duration(FixedSchema {
- name: name.clone(),
- aliases: None,
- doc: None,
- size: 12,
- attributes: Default::default(),
- });
- named_schemas.insert(name);
- schema
+ named_schemas.insert(name.clone());
+ Schema::record(name)
+ .fields(vec![
+ // Secs is an u64
+ RecordField::builder()
+ .name("secs")
+ .schema(u64::get_schema_in_ctxt(named_schemas, enclosing_namespace))
+ .build(),
+ // Nanos is an u32
+ RecordField::builder()
+ .name("nanos")
+ .schema(Schema::Long)
+ .build(),
+ ])
+ .build()
}
}
Review Comment:
`get_record_fields_in_ctxt()` should `Some(vec![records fields])` too
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
Review Comment:
```suggestion
//! - **unit variant** => See [Enums](#enums)
```
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
Review Comment:
```suggestion
//! - **newtype variant** => See [Enums](#enums)
```
##########
avro/src/serde/deser_schema/mod.rs:
##########
@@ -0,0 +1,1818 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, collections::HashMap, io::Read};
+
+use serde::de::{Deserializer, Visitor};
+
+use crate::{
+ Error, Schema,
+ decode::decode_len,
+ error::Details,
+ schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema},
+ util::{zag_i32, zag_i64},
+};
+
+mod array;
+mod enums;
+mod identifier;
+mod map;
+mod record;
+mod tuple;
+
+use array::ArrayDeserializer;
+use enums::PlainEnumDeserializer;
+use map::MapDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let variant = schema.get_variant(index as usize)?;
Review Comment:
```suggestion
let index = zag_i32(self.reader)?;
let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
let variant = schema.get_variant(index)?;
```
##########
avro/src/serde/ser_schema/block.rs:
##########
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeMap, SerializeSeq},
+};
+
+use super::{Config, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ encode::encode_int,
+ error::Details,
+ schema::{ArraySchema, MapSchema},
+ util::zig_i32,
+};
+
+#[expect(
+ private_interfaces,
+ reason = "Direct and Buffered should not be used directly"
+)]
+pub enum BlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Direct(DirectBlockSerializer<'s, 'w, W, S>),
+ Buffered(BufferedBlockSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BlockSerializer<'s, 'w, W, S> {
+ pub fn array(
+ writer: &'w mut W,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let bytes_written = bytes_written.unwrap_or(0);
+ if let Some(len) = len
+ && config.target_block_size.is_none()
+ {
+ Ok(Self::Direct(DirectBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ } else {
+ let target_block_size = config.target_block_size.unwrap_or(1024);
+ Ok(Self::Buffered(BufferedBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ target_block_size,
+ bytes_written,
+ )))
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_element(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_key(key),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_value(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+struct DirectBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> DirectBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: usize,
+ mut bytes_written: usize,
+ ) -> Result<Self, Error> {
+ if len != 0 {
+ // .end() always writes the zero block, so we only write the size for arrays
+ // that have at least one element
+ bytes_written += zig_i32(len as i32, &mut *writer)?;
+ }
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ bytes_written,
+ })
+ }
+
+ fn end(self) -> Result<usize, Error> {
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+struct BufferedBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ buffer: Vec<u8>,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+ items_in_buffer: usize,
+ target_block_size: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BufferedBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ target_block_size: usize,
+ bytes_written: usize,
+ ) -> Self {
+ Self {
+ writer,
+ buffer: Vec::with_capacity(target_block_size),
+ schema,
+ config,
+ bytes_written,
+ items_in_buffer: 0,
+ target_block_size,
+ }
+ }
+
+ /// Write a block including the items and size header.
+ fn write_block(&mut self) -> Result<(), Error> {
+ // Write the header, the negative item count indicates that the next value is the size of the
+ // block in bytes
+ self.bytes_written += encode_int(0 - (self.items_in_buffer as i32), &mut *self.writer)?;
+ self.bytes_written += encode_int(self.buffer.len() as i32, &mut *self.writer)?;
+
+ // Write the actual data
+ self.writer
+ .write_all(&self.buffer)
+ .map_err(Details::WriteBytes)?;
+ self.bytes_written += self.buffer.len();
+
+ // Reset the buffer
+ self.items_in_buffer = 0;
+ self.buffer.clear();
+
+ Ok(())
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any items remaining in the buffer
+ if self.items_in_buffer > 0 {
+ self.write_block()?;
+ }
+ debug_assert_eq!(self.buffer.len(), 0, "Buffer must be empty at this point");
+
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ value.serialize(SchemaAwareSerializer::new(
+ &mut self.buffer,
+ self.schema,
+ self.config,
+ )?)?;
+ self.items_in_buffer += 1;
+
+ if self.buffer.len() >= self.target_block_size {
+ self.write_block()?;
+ }
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ key.serialize(SchemaAwareSerializer::new(
+ &mut self.writer,
Review Comment:
```suggestion
&mut self.buffer,
```
?
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
Review Comment:
```suggestion
))
```
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
Review Comment:
```suggestion
Err(self.error("bool", "Expected Schema::Boolean in variants"))
```
##########
avro/src/serde/ser_schema/tuple.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeTuple, SerializeTupleStruct, SerializeTupleVariant},
+};
+
+use super::{Config, SchemaAwareSerializer, union::UnionSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{RecordSchema, UnionSchema},
+};
+
+#[expect(
+ private_interfaces,
+ reason = "One{,Union}TupleSerializer should not be used directly"
+)]
+pub enum TupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Unit(usize),
+ One(OneTupleSerializer<'s, 'w, W, S>),
+ /// This exists because we can't create a `&Schema::Union` from a `&UnionSchema`
+ OneUnion(OneUnionTupleSerializer<'s, 'w, W, S>),
+ Many(ManyTupleSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> TupleSerializer<'s, 'w, W, S> {
+ pub fn unit(bytes_written: Option<usize>) -> Self {
+ Self::Unit(bytes_written.unwrap_or(0))
+ }
+
+ pub fn one(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::One(OneTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+
+ pub fn one_union(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self::OneUnion(OneUnionTupleSerializer::new(writer, union, config))
+ }
+
+ pub fn many(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::Many(ManyTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for TupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ TupleSerializer::Unit(_) => Err(Error::new(Details::SerializeValueWithSchema {
+ value_type: "tuple",
+ value: "Expected no elements for the unit tuple".into(),
+ schema: Schema::Null,
+ })),
+ TupleSerializer::One(one) => one.serialize_element(value),
+ TupleSerializer::OneUnion(one) => one.serialize_element(value),
+ TupleSerializer::Many(many) => many.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ TupleSerializer::Unit(bytes_written) => Ok(bytes_written),
+ TupleSerializer::One(one) => one.end(),
+ TupleSerializer::OneUnion(one) => one.end(),
+ TupleSerializer::Many(many) => SerializeTuple::end(many),
+ }
+ }
+}
+
+pub struct ManyTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> ManyTupleSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ schema,
+ config,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for ManyTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let schema = &self.schema.fields[self.field_position].schema;
Review Comment:
Let's return an Err if `self.field_position` is out of bounds
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
+ }
+ }
+
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Double) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f64", "Expected Schema::Double in variants"))?
Review Comment:
```suggestion
Err(self.error("f64", "Expected Schema::Double in variants"))
```
##########
avro/src/error.rs:
##########
@@ -553,32 +555,51 @@ pub enum Details {
#[error("Decoded integer out of range for i32: {1}: {0}")]
ZagI32(#[source] std::num::TryFromIntError, i64),
- #[error("unable to read block")]
+ #[error("Did not read any bytes, block is corrupt")]
ReadBlock,
#[error("Failed to serialize value into Avro value: {0}")]
SerializeValue(String),
- #[error("Failed to serialize value of type {value_type} using schema {schema:?}: {value}")]
+ #[error("Failed to serialize value of type `{value_type}` using Schema::{schema:?}: {value}")]
SerializeValueWithSchema {
value_type: &'static str,
value: String,
schema: Schema,
},
- #[error("Failed to serialize field '{field_name}' for record {record_schema:?}: {error}")]
+ #[error("{position} is not a valid index for fields in {schema:?}")]
+ SerializeRecordUnknownFieldIndex {
+ position: usize,
+ schema: RecordSchema,
+ },
+
+ #[error("Failed to serialize field '{field_name}' of record {record_schema:?}: {error}")]
SerializeRecordFieldWithSchema {
field_name: String,
- record_schema: Schema,
- error: Box<Error>,
+ record_schema: RecordSchema,
+ error: String,
Review Comment:
Why the source/cause is a String now ?
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
Review Comment:
```suggestion
//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serializer::serialize_bytes`](serde::Serializer), [`Deserializer::deserialize_bytes`](serde::Deserializer), [`Deserializer::deserialize_byte_buf`](serde::Deserializer))
```
##########
avro_derive/src/attributes/mod.rs:
##########
@@ -340,6 +340,12 @@ impl FieldOptions {
return Err(errors);
}
+ // TODO: Implement a better way to do this (maybe if user specifies `#[avro(with)]` also use that for the default)
+ // Disable getting the field default, if the schema is not retrieved from the field type
+ if with != With::Trait && avro.default == FieldDefault::Trait {
+ avro.default = FieldDefault::Disabled;
Review Comment:
Should this check be made before the `skip_serializing` check at line 321 ?
##########
avro/src/serde/ser_schema/union.rs:
##########
@@ -0,0 +1,566 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{Serialize, Serializer};
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{FixedSchema, SchemaKind, UnionSchema},
+ serde::{
+ ser_schema::{
+ block::BlockSerializer,
+ record::RecordSerializer,
+ tuple::{ManyTupleSerializer, TupleSerializer},
+ },
+ with::{BytesType, SER_BYTES_TYPE},
+ },
+ util::{zig_i32, zig_i64},
+};
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 8, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
+ }
+ }
+
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Double) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f64", "Expected Schema::Double in variants"))?
+ }
+ }
+
+ fn serialize_char(mut self, v: char) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.to_string().as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("char", "Expected Schema::String in variants"))?
Review Comment:
```suggestion
Err(self.error("char", "Expected Schema::String in variants"))
```
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol must match the index of the variant
+//! - `arrays`: [`Vec`] (or any type that uses [`Serialize::serialize_seq`](serde::Serialize), [`Deserialize::deserialize_seq`](serde::Deserialize))
Review Comment:
```suggestion
//! - `arrays`: [`Vec`] (or any type that uses [`Serializer::serialize_seq`](serde::Serializer), [`Deserializer::deserialize_seq`](serde::Deserializer))
```
##########
avro/src/schema/mod.rs:
##########
@@ -5118,4 +5183,29 @@ mod tests {
Ok(())
}
+
+ #[test]
+ fn avro_rs_512_unique_normalized_name_must_be_unique() -> TestResult {
+ let one = Schema::Ref {
+ name: "a.b.c".parse()?,
+ };
+ let two = Schema::Ref {
+ name: "a_b_c".parse()?,
+ };
+ let three = Schema::Ref {
+ name: "a__b__c".parse()?,
+ };
+ let four = Schema::Ref {
+ name: "a._b._c".parse()?,
+ };
+
+ assert_ne!(one, two);
+ assert_ne!(one, three);
+ assert_ne!(one, four);
+ assert_ne!(two, three);
+ assert_ne!(two, four);
+ assert_ne!(three, four);
Review Comment:
```suggestion
assert_ne!(one.unique_normalized_name(), two.unique_normalized_name());
assert_ne!(one.unique_normalized_name(), three.unique_normalized_name());
assert_ne!(one.unique_normalized_name(), four.unique_normalized_name());
assert_ne!(two.unique_normalized_name(), three.unique_normalized_name());
assert_ne!(two.unique_normalized_name(), four.unique_normalized_name());
assert_ne!(three.unique_normalized_name(), four.unique_normalized_name());
```
##########
avro/src/documentation/serde_data_model_to_avro.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. When mapping from an Avro schema to Rust types,
+//! see [the documentation for the reverse](super::avro_data_model_to_serde).
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema of the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples`.
+//! - **tuple_variant** => See [Enums](##Enums)
+//! - **map** => [`Schema::Map`]
+//! - **Note:** the key type of the map will be (de)serialized as a [`Schema::String`]
+//! - **struct** => [`Schema::Record`]
+//! - **struct_variant** => See [Enums](##Enums)
Review Comment:
```suggestion
//! - **struct_variant** => See [Enums](#enums)
```
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol must match the index of the variant
+//! - `arrays`: [`Vec`] (or any type that uses [`Serialize::serialize_seq`](serde::Serialize), [`Deserialize::deserialize_seq`](serde::Deserialize))
+//! - `[T; N]` is (de)serialized as a tuple by Serde, to (de)serialize them as an `array` use [`apache_avro::serde::array`]
+//! - `maps`: [`HashMap<String, _>`](std::collections::HashMap) (or any type that uses [`Serialize::serialize_map`](serde::Serialize), [`Deserialize::deserialize_map`](serde::Deserialize))
Review Comment:
```suggestion
//! - `maps`: [`HashMap<String, _>`](std::collections::HashMap) (or any type that uses [`Serializer::serialize_map`](serde::Serializer), [`Deserializer::deserialize_map`](serde::Deserializer))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2969321379
##########
avro/src/serde/ser_schema/tuple.rs:
##########
@@ -0,0 +1,283 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{borrow::Borrow, io::Write};
+
+use serde::{
+ Serialize,
+ ser::{SerializeTuple, SerializeTupleStruct, SerializeTupleVariant},
+};
+
+use super::{Config, SchemaAwareSerializer, union::UnionSerializer};
+use crate::{
+ Error, Schema,
+ error::Details,
+ schema::{RecordSchema, UnionSchema},
+};
+
+#[expect(
+ private_interfaces,
+ reason = "One{,Union}TupleSerializer should not be used directly"
+)]
+pub enum TupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Unit(usize),
+ One(OneTupleSerializer<'s, 'w, W, S>),
+ /// This exists because we can't create a `&Schema::Union` from a `&UnionSchema`
+ OneUnion(OneUnionTupleSerializer<'s, 'w, W, S>),
+ Many(ManyTupleSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> TupleSerializer<'s, 'w, W, S> {
+ pub fn unit(bytes_written: Option<usize>) -> Self {
+ Self::Unit(bytes_written.unwrap_or(0))
+ }
+
+ pub fn one(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::One(OneTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+
+ pub fn one_union(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self::OneUnion(OneUnionTupleSerializer::new(writer, union, config))
+ }
+
+ pub fn many(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::Many(ManyTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for TupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ TupleSerializer::Unit(_) => Err(Error::new(Details::SerializeValueWithSchema {
+ value_type: "tuple",
+ value: "Expected no elements for the unit tuple".into(),
+ schema: Schema::Null,
+ })),
+ TupleSerializer::One(one) => one.serialize_element(value),
+ TupleSerializer::OneUnion(one) => one.serialize_element(value),
+ TupleSerializer::Many(many) => many.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ TupleSerializer::Unit(bytes_written) => Ok(bytes_written),
+ TupleSerializer::One(one) => one.end(),
+ TupleSerializer::OneUnion(one) => one.end(),
+ TupleSerializer::Many(many) => SerializeTuple::end(many),
+ }
+ }
+}
+
+pub struct ManyTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> ManyTupleSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ schema,
+ config,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for ManyTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let schema = &self.schema.fields[self.field_position].schema;
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ schema,
+ self.config,
+ )?)?;
+ self.field_position += 1;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ assert_eq!(self.field_position, self.schema.fields.len());
Review Comment:
Converted to an error, originally chose a panic because this can only happen for invalid Serialize implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2966012621
##########
avro/src/documentation/avro_data_model_to_serde.rs:
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Avro data model to the Serde data model
+//!
+//! When manually mapping an Avro schema to Rust types it is important to understand
+//! how the different data models are mapped. When mapping from Rust types to an Avro schema,
+//! see [the documentation for the reverse](super::serde_data_model_to_avro).
+//!
+//! Only the the mapping as defined here is supported. Any other behavior might change in a minor version.
+//!
+//! ## Primitive Types
+//! - `null`: `()`
+//! - `boolean`: [`bool`]
+//! - `int`: [`i32`] (or [`i16`], [`i8`], [`u16`], [`u8`])
+//! - `long`: [`i64`] (or [`u32`])
+//! - `float`: [`f32`]
+//! - `double`: [`f64`]
+//! - `bytes`: [`Vec<u8>`](std::vec::Vec) (or any type that uses [`Serialize::serialize_bytes`](serde::Serialize), [`Deserialize::deserialize_bytes`](serde::Deserialize), [`Deserialize::deserialize_byte_buf`](serde::Deserialize))
+//! - It is required to use [`apache_avro::serde::bytes`] as otherwise Serde will (de)serialize a `Vec` as an array of integers instead.
+//! - `string`: [`String`] (or any type that uses [`Serialize::serialize_str`](serde::Serialize), [`Deserialize::deserialize_str`](serde::Deserialize), [`Deserialize::deserialize_string`](serde::Deserialize))
+//!
+//! ## Complex Types
+//! - `records`: A struct with the same name and fields or a tuple with the same fields.
+//! - Extra fields can be added to the struct if they are marked with `#[serde(skip)]`
+//! - `enums`: A enum with the same name and unit variants for every symbol
+//! - The index of the symbol most match the index of the variant
+//! - `arrays`: [`Vec`] (or any type that uses [`Serialize::serialize_seq`](serde::Serialize), [`Deserialize::deserialize_seq`](serde::Deserialize))
+//! - `[T; N]` is (de)serialized as a tuple by Serde, to (de)serialize them as an `array` use [`apache_avro::serde::array`]
+//! - `maps`: [`HashMap<String, _>`](std::collections::HashMap) (or any type that uses [`Serialize::serialize_map`](serde::Serialize), [`Deserialize::deserialize_map`](serde::Deserialize))
+//! - `unions`: A enum with a variant for each variant
Review Comment:
```suggestion
//! - `unions`: An enum with a variant for each union variant
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r2980248288
##########
avro/src/serde/ser_schema/record/field_default.rs:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+use crate::{
+ Schema,
+ schema::{DecimalSchema, InnerDecimalSchema, SchemaKind, UnionSchema, UuidSchema},
+ serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT,
+};
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (
+ Value::String(_),
+ Schema::Bytes
+ | Schema::String
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ ..
+ })
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::String),
+ ) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.as_f64().is_some() => true,
+ (
+ Value::String(s),
+ Schema::Fixed(fixed)
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(fixed),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(fixed))
+ | Schema::Duration(fixed),
+ ) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.as_f64().is_some() => {
+ serializer.serialize_f32(n.as_f64().unwrap() as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.as_f64().is_some() => {
+ serializer.serialize_f64(n.as_f64().unwrap())
+ }
+ (
+ Value::String(s),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_))
+ | Schema::BigDecimal
+ | Schema::Decimal(_)
+ | Schema::Duration(_)
+ | Schema::Date,
Review Comment:
Confused it with Schema::Duration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on code in PR #512:
URL: https://github.com/apache/avro-rs/pull/512#discussion_r3032068766
##########
avro/src/reader/single_object.rs:
##########
@@ -68,6 +79,26 @@ impl GenericSingleObjectReader {
Err(io_error) => Err(Details::ReadHeader(io_error).into()),
}
}
+
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ let mut header = vec![0; self.expected_header.len()];
Review Comment:
This simplifies the code significantly, good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-4152673765
@martin-g anything blocking this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat!: Support enums and tuples in `SchemaAwareSerializer` and implement `SchemaAwareDeserializer` [avro-rs]
Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on PR #512:
URL: https://github.com/apache/avro-rs/pull/512#issuecomment-5553786557
@kevinjqliu that's great to hear! Please do let me know if you run into any issues or find anything lacking in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org