You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@avro.apache.org by kr...@apache.org on 2026/03/19 12:56:00 UTC
(avro-rs) branch feat/full_enum_support updated (f16fc52 -> c3e1825)
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a change to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
discard f16fc52 temp
discard 5e72d86 feat: Rework `SchemaAwareSerializer` to be more strict and follow the documented model (no tests)
discard 479ff57 feat: Use `SchemaAwareDeserializer` in the readers and add tests
discard c3e1848 feat: `SchemaAwareDeserializer` (no tests)
discard de8cced doc: Document the mapping between the Serde and Avro data models
add 1452bf7 chore(deps): Bump bon from 3.9.0 to 3.9.1 (#511)
new 3d33135 doc: Document the mapping between the Serde and Avro data models
new 764f119 feat: `SchemaAwareDeserializer` (no tests)
new 2317898 feat: Use `SchemaAwareDeserializer` in the readers and add tests
new 4da2ebf feat: Rework `SchemaAwareSerializer` to be more strict and follow the documented model (no tests)
new c3e1825 temp
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f16fc52)
\
N -- N -- N refs/heads/feat/full_enum_support (c3e1825)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Cargo.lock | 8 ++++----
avro/Cargo.toml | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
(avro-rs) 01/05: doc: Document the mapping between the Serde and Avro data models
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a commit to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
commit 3d331350a1e4d4214a7cefd8aa4d924bfae3e75a
Author: Kriskras99 <gi...@kriskras99.nl>
AuthorDate: Fri Mar 13 11:25:58 2026 +0100
doc: Document the mapping between the Serde and Avro data models
---
avro/src/documentation/mod.rs | 1 +
avro/src/documentation/serde_data_model_to_avro.rs | 116 +++++++++++++++++++++
2 files changed, 117 insertions(+)
diff --git a/avro/src/documentation/mod.rs b/avro/src/documentation/mod.rs
index 7771012..dcd0848 100644
--- a/avro/src/documentation/mod.rs
+++ b/avro/src/documentation/mod.rs
@@ -22,3 +22,4 @@
pub mod dynamic;
pub mod primer;
+pub mod serde_data_model_to_avro;
diff --git a/avro/src/documentation/serde_data_model_to_avro.rs b/avro/src/documentation/serde_data_model_to_avro.rs
new file mode 100644
index 0000000..b623577
--- /dev/null
+++ b/avro/src/documentation/serde_data_model_to_avro.rs
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! # Mapping the Serde data model to the Avro data model
+//!
+//! When manually mapping Rust types to an Avro schema, or the reverse, it is important to understand
+//! how the different data models are mapped. This should only be done in very specific circumstances,
+//! most users should use the [`AvroSchema`] derive.
+//!
+//! Only the schemas generated by the [`AvroSchema`] derive and the mapping as defined here are
+//! supported. Any other behavior might change in a minor version.
+//!
+//! The following list is based on [the data model defined by Serde](https://serde.rs/data-model.html):
+//! - **14 primitive types**
+//! - `bool` => [`Schema::Boolean`]
+//! - `i8`, `i16`, `i32`, `u8`, `u16` => [`Schema::Int`]
+//! - `i64`, `u32` => [`Schema::Long`]
+//! - `u64` => [`Schema::Fixed`]`(name: "u64", size: 8)`
+//! - This is not a `Schema::Long` as that is a signed number of maximum 64 bits.
+//! - `i128` => [`Schema::Fixed`]`(name: "i128", size: 16)`
+//! - `u128` => [`Schema::Fixed`]`(name: "u128", size: 16)`
+//! - `f32` => [`Schema::Float`]
+//! - `f64` => [`Schema::Double`]
+//! - `char` => [`Schema::String`]
+//! - Only one character allowed, deserializer will return an error for strings with more than one character.
+//! - **string** => [`Schema::String`]
+//! - **byte array** => [`Schema::Bytes`] or [`Schema::Fixed`]
+//! - **option** => [`Schema::Union([Schema::Null, _])`](crate::schema::Schema::Union)
+//! - **unit** => [`Schema::Null`]
+//! - **unit struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and zero fields
+//! - **unit variant** => See [Enums](##Enums)
+//! - **newtype struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and one field
+//! - **newtype variant** => See [Enums](##Enums)
+//! - **seq** => [`Schema::Array`]
+//! - **tuple**
+//! - For tuples with one element, the schema will be the schema the only element
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
+//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
+//! [`Schema::Array`] use [`apache_avro::serde::array`].
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
+//! is different from normal tuples`.
+//! - **tuple_variant** => See [Enums](##Enums)
+//! - **map** => [`Schema::Map`]
+//! - **Note:** the key type of the map will be (de)serialized as a [`Schema::String`]
+//! - **struct** => [`Schema::Record`]
+//! - **struct_variant** => See [Enums](##Enums)
+//!
+//! ## Enums
+//!
+//! ### Externally tagged
+//! This is the default enum representation for Serde. It can be mapped in three ways to the Avro data model.
+//! For all three options it is important that the enum index matches the Avro index.
+//! - As a [`Schema::Enum`], this is only possible for enums with only unit variants.
+//! - As a [`Schema::Union`] with a [`Schema::Record`] for every variant:
+//! - **unit_variant** => [`Schema::Record`] named as the variant and with no fields.
+//! - **newtype_variant** => [`Schema::Record`] named as the variant and with one field.
+//! The schema must have the attribute `org.apache.avro.rust.union_of_records` with the value set to `true`.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//! - As a [`Schema::Union`] without the wrapper [`Schema::Record`], all schemas must be unique:
+//! - **unit_variant** => [`Schema::Null`].
+//! - **newtype_variant** => The schema of the inner type.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//!
+//! ### Internally tagged
+//! This enum representation is used by Serde if the attribute `#[serde(tag = "...")]` is used.
+//! It maps to a [`Schema::Record`]. There must be at least one field that is named as the value of the
+//! `tag` attribute. If a field is not used by all variants it must have a `default` set.
+//!
+//! ### Adjacently tagged
+//! This enum representation is used by Serde if the attributes `#[serde(tag = "...", content = "...")]` are used.
+//! It maps to a [`Schema::Record`] with two fields. One field must be named as the value of the `tag`
+//! attribute and use the [`Schema::Enum`] schema. The other field must be named as the value of the
+//! `content` tag and use the [`Schema::Union`] schema.
+//!
+//! ### Untagged
+//! This enum representation is ued by Serde if the attribute `#[serde(untagged)]` is used. It maps
+//! to a [`Schema::Union`] with the following schemas:
+//! - **unit_variant** => [`Schema::Null`].
+//! - **newtype_variant** => The schema of the inner type.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//!
+//! [`AvroSchema`]: crate::AvroSchema
+//! [`Schema::Array`]: crate::schema::Schema::Array
+//! [`Schema::Boolean`]: crate::schema::Schema::Boolean
+//! [`Schema::Bytes`]: crate::schema::Schema::Bytes
+//! [`Schema::Double`]: crate::schema::Schema::Double
+//! [`Schema::Enum`]: crate::schema::Schema::Enum
+//! [`Schema::Fixed`]: crate::schema::Schema::Fixed
+//! [`Schema::Float`]: crate::schema::Schema::Float
+//! [`Schema::Int`]: crate::schema::Schema::Int
+//! [`Schema::Long`]: crate::schema::Schema::Long
+//! [`Schema::Map`]: crate::schema::Schema::Map
+//! [`Schema::Null`]: crate::schema::Schema::Null
+//! [`Schema::Record`]: crate::schema::Schema::Record
+//! [`Schema::String`]: crate::schema::Schema::String
+//! [`Schema::Union`]: crate::schema::Schema::Union
+//! [`apache_avro::serde::array`]: crate::serde::array
(avro-rs) 03/05: feat: Use `SchemaAwareDeserializer` in the readers and add tests
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a commit to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
commit 2317898c125a115787eb7b320ca817f675372403
Author: Kriskras99 <gi...@kriskras99.nl>
AuthorDate: Fri Mar 13 20:00:50 2026 +0100
feat: Use `SchemaAwareDeserializer` in the readers and add tests
---
.../test_interop_single_object_encoding.rs | 4 +-
avro/src/reader/block.rs | 47 +-
avro/src/reader/datum.rs | 111 ++-
avro/src/reader/mod.rs | 44 +-
avro/src/reader/single_object.rs | 82 +-
avro/src/serde/deser_schema/mod.rs | 949 +++++++++++++++++++++
6 files changed, 1186 insertions(+), 51 deletions(-)
diff --git a/avro/examples/test_interop_single_object_encoding.rs b/avro/examples/test_interop_single_object_encoding.rs
index e16c221..5b8e1bb 100644
--- a/avro/examples/test_interop_single_object_encoding.rs
+++ b/avro/examples/test_interop_single_object_encoding.rs
@@ -74,7 +74,9 @@ fn test_write(expected: &[u8]) {
fn test_read(encoded: Vec<u8>) {
let mut encoded = &encoded[..];
- let read_message = apache_avro::GenericSingleObjectReader::new(InteropMessage::get_schema())
+ let read_message = apache_avro::GenericSingleObjectReader::builder()
+ .schema(InteropMessage::get_schema())
+ .build()
.expect("Resolving failed")
.read_value(&mut encoded)
.expect("Decoding failed");
diff --git a/avro/src/reader/block.rs b/avro/src/reader/block.rs
index 06c07a2..1d80871 100644
--- a/avro/src/reader/block.rs
+++ b/avro/src/reader/block.rs
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
use crate::{
AvroResult, Codec, Error,
decode::{decode, decode_internal},
@@ -24,6 +25,7 @@ use crate::{
util,
};
use log::warn;
+use serde::de::DeserializeOwned;
use serde_json::from_slice;
use std::{
collections::HashMap,
@@ -46,10 +48,15 @@ pub(super) struct Block<'r, R> {
schemata: Vec<&'r Schema>,
pub(super) user_metadata: HashMap<String, Vec<u8>>,
names_refs: Names,
+ human_readable: bool,
}
impl<'r, R: Read> Block<'r, R> {
- pub(super) fn new(reader: R, schemata: Vec<&'r Schema>) -> AvroResult<Block<'r, R>> {
+ pub(super) fn new(
+ reader: R,
+ schemata: Vec<&'r Schema>,
+ human_readable: bool,
+ ) -> AvroResult<Block<'r, R>> {
let mut block = Block {
reader,
codec: Codec::Null,
@@ -61,6 +68,7 @@ impl<'r, R: Read> Block<'r, R> {
marker: [0; 16],
user_metadata: Default::default(),
names_refs: Default::default(),
+ human_readable,
};
block.read_header()?;
@@ -205,6 +213,43 @@ impl<'r, R: Read> Block<'r, R> {
Ok(Some(item))
}
+ pub(super) fn read_next_deser<T: DeserializeOwned>(
+ &mut self,
+ read_schema: Option<&Schema>,
+ ) -> AvroResult<Option<T>> {
+ if self.is_empty() {
+ self.read_block_next()?;
+ if self.is_empty() {
+ return Ok(None);
+ }
+ }
+
+ let mut block_bytes = &self.buf[self.buf_idx..];
+ let b_original = block_bytes.len();
+
+ let item = if read_schema.is_some() {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
+ } else {
+ let config = Config {
+ names: &self.names_refs,
+ human_readable: self.human_readable,
+ };
+ T::deserialize(SchemaAwareDeserializer::new(
+ &mut block_bytes,
+ &self.writer_schema,
+ config,
+ )?)?
+ };
+
+ if b_original != 0 && b_original == block_bytes.len() {
+ // No bytes were read, return an error to avoid an infinite loop
+ return Err(Details::ReadBlock.into());
+ }
+ self.buf_idx += b_original - block_bytes.len();
+ self.message_count -= 1;
+ Ok(Some(item))
+ }
+
fn read_writer_schema(&mut self, metadata: &HashMap<String, Value>) -> AvroResult<()> {
let json: serde_json::Value = metadata
.get("avro.schema")
diff --git a/avro/src/reader/datum.rs b/avro/src/reader/datum.rs
index 80fe937..6f5906f 100644
--- a/avro/src/reader/datum.rs
+++ b/avro/src/reader/datum.rs
@@ -15,11 +15,16 @@
// specific language governing permissions and limitations
// under the License.
-use std::io::Read;
-
+use crate::schema::ResolvedOwnedSchema;
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
+use crate::util::is_human_readable;
+use crate::{
+ AvroResult, AvroSchema, Schema, decode::decode_internal, schema::ResolvedSchema, types::Value,
+};
use bon::bon;
-
-use crate::{AvroResult, Schema, decode::decode_internal, schema::ResolvedSchema, types::Value};
+use serde::de::DeserializeOwned;
+use std::io::Read;
+use std::marker::PhantomData;
/// Reader for reading raw Avro data.
///
@@ -30,6 +35,7 @@ pub struct GenericDatumReader<'s> {
writer: &'s Schema,
resolved: ResolvedSchema<'s>,
reader: Option<(&'s Schema, ResolvedSchema<'s>)>,
+ human_readable: bool,
}
#[bon]
@@ -50,6 +56,9 @@ impl<'s> GenericDatumReader<'s> {
reader_schema: Option<&'s Schema>,
/// Already resolved schemata that will be used to resolve references in the reader's schema.
resolved_reader_schemata: Option<ResolvedSchema<'s>>,
+ /// Was the data serialized with `human_readable`.
+ #[builder(default = is_human_readable())]
+ human_readable: bool,
) -> AvroResult<Self> {
let resolved_writer_schemata = if let Some(resolved) = resolved_writer_schemata {
resolved
@@ -71,6 +80,7 @@ impl<'s> GenericDatumReader<'s> {
writer: writer_schema,
resolved: resolved_writer_schemata,
reader,
+ human_readable,
})
}
}
@@ -124,6 +134,68 @@ impl<'s> GenericDatumReader<'s> {
Ok(value)
}
}
+
+ /// Read a Avro datum from the reader.
+ ///
+ /// # Panics
+ /// Will panic if a reader schema has been configured, this is a WIP.
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ // `reader` is `impl Read` instead of a generic on the function like T so it's easier to
+ // specify the type wanted (`read_deser<String>` vs `read_deser<String, _>`)
+ if let Some((_, _)) = &self.reader {
+ todo!("Schema aware deserialisation does not resolve schemas yet");
+ } else {
+ T::deserialize(SchemaAwareDeserializer::new(
+ reader,
+ self.writer,
+ Config {
+ names: self.resolved.get_names(),
+ human_readable: self.human_readable,
+ },
+ )?)
+ }
+ }
+}
+
+/// Reader for reading raw Avro data.
+///
+/// This is most likely not what you need. Most users should use [`Reader`][crate::Reader],
+/// [`GenericSingleObjectReader`][crate::GenericSingleObjectReader], or
+/// [`SpecificSingleObjectReader`][crate::SpecificSingleObjectReader] instead.
+pub struct SpecificDatumReader<T: AvroSchema> {
+ resolved: ResolvedOwnedSchema,
+ human_readable: bool,
+ phantom: PhantomData<T>,
+}
+
+#[bon]
+impl<T: AvroSchema> SpecificDatumReader<T> {
+ /// Build a [`SpecificDatumReader`].
+ ///
+ /// This is most likely not what you need. Most users should use [`Reader`][crate::Reader],
+ /// [`GenericSingleObjectReader`][crate::GenericSingleObjectReader], or
+ /// [`SpecificSingleObjectReader`][crate::SpecificSingleObjectReader] instead.
+ #[builder]
+ pub fn new(#[builder(default = is_human_readable())] human_readable: bool) -> AvroResult<Self> {
+ Ok(Self {
+ resolved: T::get_schema().try_into()?,
+ human_readable,
+ phantom: PhantomData,
+ })
+ }
+}
+
+impl<T: AvroSchema + DeserializeOwned> SpecificDatumReader<T> {
+ pub fn read<R: Read>(&self, reader: &mut R) -> AvroResult<T> {
+ T::deserialize(SchemaAwareDeserializer::new(
+ reader,
+ self.resolved.get_root_schema(),
+ Config {
+ names: self.resolved.get_names(),
+ human_readable: self.human_readable,
+ },
+ )?)
+ }
}
/// Deprecated.
@@ -227,7 +299,7 @@ mod tests {
use serde::Deserialize;
use crate::{
- Schema, from_value,
+ Schema,
reader::datum::GenericDatumReader,
types::{Record, Value},
};
@@ -272,7 +344,7 @@ mod tests {
const TEST_RECORD_SCHEMA_3240: &str = r#"
{
"type": "record",
- "name": "test",
+ "name": "TestRecord3240",
"fields": [
{
"name": "a",
@@ -312,26 +384,17 @@ mod tests {
}
let schema = Schema::parse_str(TEST_RECORD_SCHEMA_3240)?;
- let mut encoded: &'static [u8] = &[54, 6, 102, 111, 111];
-
- let expected_record: TestRecord3240 = TestRecord3240 {
- a: 27i64,
- b: String::from("foo"),
- a_nullable_array: None,
- a_nullable_string: None,
- };
+ let mut encoded: &[u8] = &[54, 6, 102, 111, 111];
- let avro_datum = GenericDatumReader::builder(&schema)
+ let error = GenericDatumReader::builder(&schema)
.build()?
- .read_value(&mut encoded)?;
- let parsed_record: TestRecord3240 = match &avro_datum {
- Value::Record(_) => from_value::<TestRecord3240>(&avro_datum)?,
- unexpected => {
- panic!("could not map avro data to struct, found unexpected: {unexpected:?}")
- }
- };
-
- assert_eq!(parsed_record, expected_record);
+ .read_deser::<TestRecord3240>(&mut encoded)
+ .unwrap_err();
+ // TODO: Create a version of this test that does schema resolution
+ assert_eq!(
+ error.to_string(),
+ "Failed to read bytes for decoding variable length integer: failed to fill whole buffer"
+ );
Ok(())
}
diff --git a/avro/src/reader/mod.rs b/avro/src/reader/mod.rs
index 94af909..072f2df 100644
--- a/avro/src/reader/mod.rs
+++ b/avro/src/reader/mod.rs
@@ -21,9 +21,12 @@ mod block;
pub mod datum;
pub mod single_object;
+use crate::util::is_human_readable;
use crate::{AvroResult, schema::Schema, types::Value};
use block::Block;
use bon::bon;
+use serde::de::DeserializeOwned;
+use std::marker::PhantomData;
use std::{collections::HashMap, io::Read};
/// Main interface for reading Avro formatted values.
@@ -68,11 +71,12 @@ impl<'a, R: Read> Reader<'a, R> {
#[builder(start_fn)] reader: R,
reader_schema: Option<&'a Schema>,
schemata: Option<Vec<&'a Schema>>,
+ #[builder(default = is_human_readable())] human_readable: bool,
) -> AvroResult<Reader<'a, R>> {
let schemata =
schemata.unwrap_or_else(|| reader_schema.map(|rs| vec![rs]).unwrap_or_default());
- let block = Block::new(reader, schemata)?;
+ let block = Block::new(reader, schemata, human_readable)?;
let mut reader = Reader {
block,
reader_schema,
@@ -97,12 +101,20 @@ impl<'a, R: Read> Reader<'a, R> {
self.reader_schema
}
- /// Get a reference to the user metadata
+ /// Get a reference to the user metadata.
#[inline]
pub fn user_metadata(&self) -> &HashMap<String, Vec<u8>> {
&self.block.user_metadata
}
+ /// Convert this reader into an iterator that deserializes to `T`.
+ pub fn into_deser_iter<T: DeserializeOwned>(self) -> ReaderDeser<'a, R, T> {
+ ReaderDeser {
+ inner: self,
+ phantom: PhantomData,
+ }
+ }
+
#[inline]
fn read_next(&mut self) -> AvroResult<Option<Value>> {
let read_schema = if self.should_resolve_schema {
@@ -113,6 +125,10 @@ impl<'a, R: Read> Reader<'a, R> {
self.block.read_next(read_schema)
}
+
+ fn read_next_deser<T: DeserializeOwned>(&mut self) -> AvroResult<Option<T>> {
+ self.block.read_next_deser(self.reader_schema)
+ }
}
impl<R: Read> Iterator for Reader<'_, R> {
@@ -133,6 +149,30 @@ impl<R: Read> Iterator for Reader<'_, R> {
}
}
+/// Wrapper around [`Reader`] where the iterator deserializes `T`.
+pub struct ReaderDeser<'a, R, T> {
+ inner: Reader<'a, R>,
+ phantom: PhantomData<T>,
+}
+
+impl<R: Read, T: DeserializeOwned> Iterator for ReaderDeser<'_, R, T> {
+ type Item = AvroResult<T>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ // Don't continue when we've errored before
+ if self.inner.errored {
+ return None;
+ }
+ match self.inner.read_next_deser::<T>() {
+ Ok(opt) => opt.map(Ok),
+ Err(e) => {
+ self.inner.errored = true;
+ Some(Err(e))
+ }
+ }
+ }
+}
+
/// Reads the marker bytes from Avro bytes generated earlier by a `Writer`
pub fn read_marker(bytes: &[u8]) -> [u8; 16] {
assert!(
diff --git a/avro/src/reader/single_object.rs b/avro/src/reader/single_object.rs
index c6151a3..47e002c 100644
--- a/avro/src/reader/single_object.rs
+++ b/avro/src/reader/single_object.rs
@@ -19,8 +19,11 @@ use crate::decode::decode_internal;
use crate::error::Details;
use crate::headers::{HeaderBuilder, RabinFingerprintHeader};
use crate::schema::ResolvedOwnedSchema;
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
use crate::types::Value;
-use crate::{AvroResult, AvroSchema, Schema, from_value};
+use crate::util::is_human_readable;
+use crate::{AvroResult, AvroSchema, Schema};
+use bon::bon;
use serde::de::DeserializeOwned;
use std::io::Read;
use std::marker::PhantomData;
@@ -28,25 +31,30 @@ use std::marker::PhantomData;
pub struct GenericSingleObjectReader {
write_schema: ResolvedOwnedSchema,
expected_header: Vec<u8>,
+ human_readable: bool,
}
+#[bon]
impl GenericSingleObjectReader {
- pub fn new(schema: Schema) -> AvroResult<GenericSingleObjectReader> {
- let header_builder = RabinFingerprintHeader::from_schema(&schema);
- Self::new_with_header_builder(schema, header_builder)
- }
-
- pub fn new_with_header_builder<HB: HeaderBuilder>(
+ #[builder]
+ pub fn new(
schema: Schema,
- header_builder: HB,
+ /// The expected header.
+ #[builder(default = RabinFingerprintHeader::from_schema(&schema).build_header())]
+ header: Vec<u8>,
+ /// Was the data serialized with `human_readable`.
+ #[builder(default = is_human_readable())]
+ human_readable: bool,
) -> AvroResult<GenericSingleObjectReader> {
- let expected_header = header_builder.build_header();
- Ok(GenericSingleObjectReader {
- write_schema: ResolvedOwnedSchema::try_from(schema)?,
- expected_header,
+ Ok(Self {
+ write_schema: schema.try_into()?,
+ expected_header: header,
+ human_readable,
})
}
+}
+impl GenericSingleObjectReader {
pub fn read_value<R: Read>(&self, reader: &mut R) -> AvroResult<Value> {
let mut header = vec![0; self.expected_header.len()];
match reader.read_exact(&mut header) {
@@ -68,6 +76,26 @@ impl GenericSingleObjectReader {
Err(io_error) => Err(Details::ReadHeader(io_error).into()),
}
}
+
+ pub fn read_deser<T: DeserializeOwned>(&self, reader: &mut impl Read) -> AvroResult<T> {
+ let mut header = vec![0; self.expected_header.len()];
+ reader
+ .read_exact(&mut header)
+ .map_err(Details::ReadHeader)?;
+ if self.expected_header == header {
+ let config = Config {
+ names: self.write_schema.get_names(),
+ human_readable: self.human_readable,
+ };
+ T::deserialize(SchemaAwareDeserializer::new(
+ reader,
+ self.write_schema.get_root_schema(),
+ config,
+ )?)
+ } else {
+ Err(Details::SingleObjectHeaderMismatch(self.expected_header.clone(), header).into())
+ }
+ }
}
pub struct SpecificSingleObjectReader<T>
@@ -84,7 +112,9 @@ where
{
pub fn new() -> AvroResult<SpecificSingleObjectReader<T>> {
Ok(SpecificSingleObjectReader {
- inner: GenericSingleObjectReader::new(T::get_schema())?,
+ inner: GenericSingleObjectReader::builder()
+ .schema(T::get_schema())
+ .build()?,
_model: PhantomData,
})
}
@@ -104,7 +134,7 @@ where
T: AvroSchema + DeserializeOwned,
{
pub fn read<R: Read>(&self, reader: &mut R) -> AvroResult<T> {
- from_value::<T>(&self.inner.read_value(reader)?)
+ self.inner.read_deser(reader)
}
}
@@ -131,7 +161,7 @@ mod tests {
let schema = r#"
{
"type":"record",
- "name":"TestSingleObjectWrtierSerialize",
+ "name":"TestSingleObjectReader",
"fields":[
{
"name":"a",
@@ -220,7 +250,9 @@ mod tests {
)
.expect("Encode should succeed");
let mut to_read = &to_read[..];
- let generic_reader = GenericSingleObjectReader::new(TestSingleObjectReader::get_schema())
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .build()
.expect("Schema should resolve");
let val = generic_reader
.read_value(&mut to_read)
@@ -254,7 +286,9 @@ mod tests {
)
.expect("Encode should succeed");
let mut to_read = (&to_read_1[..]).chain(&to_read_2[..]).chain(&to_read_3[..]);
- let generic_reader = GenericSingleObjectReader::new(TestSingleObjectReader::get_schema())
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .build()
.expect("Schema should resolve");
let val = generic_reader
.read_value(&mut to_read)
@@ -286,7 +320,9 @@ mod tests {
&mut to_read,
)
.expect("Encode should succeed");
- let generic_reader = GenericSingleObjectReader::new(TestSingleObjectReader::get_schema())
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .build()
.expect("Schema should resolve");
let specific_reader = SpecificSingleObjectReader::<TestSingleObjectReader>::new()
.expect("schema should resolve");
@@ -315,11 +351,11 @@ mod tests {
fn avro_rs_164_generic_reader_alternate_header() -> TestResult {
let schema_uuid = Uuid::parse_str("b2f1cf00-0434-013e-439a-125eb8485a5f")?;
let header_builder = GlueSchemaUuidHeader::from_uuid(schema_uuid);
- let generic_reader = GenericSingleObjectReader::new_with_header_builder(
- TestSingleObjectReader::get_schema(),
- header_builder,
- )
- .expect("failed to build reader");
+ let generic_reader = GenericSingleObjectReader::builder()
+ .schema(TestSingleObjectReader::get_schema())
+ .header(header_builder.build_header())
+ .build()
+ .expect("failed to build reader");
let data_to_read: Vec<u8> = vec![
3, 0, 178, 241, 207, 0, 4, 52, 1, 62, 67, 154, 18, 94, 184, 72, 90, 95,
];
diff --git a/avro/src/serde/deser_schema/mod.rs b/avro/src/serde/deser_schema/mod.rs
index 99d5897..f7c8dc6 100644
--- a/avro/src/serde/deser_schema/mod.rs
+++ b/avro/src/serde/deser_schema/mod.rs
@@ -739,3 +739,952 @@ impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
self.config.human_readable
}
}
+
+#[cfg(test)]
+mod tests {
+ use std::fmt::Debug;
+ use std::io::Cursor;
+
+ use num_bigint::BigInt;
+ use pretty_assertions::assert_eq;
+ use serde::de::{DeserializeOwned, Visitor};
+ use serde::{Deserialize, Serialize};
+ use uuid::Uuid;
+
+ use apache_avro_test_helper::TestResult;
+
+ use super::*;
+ use crate::reader::datum::GenericDatumReader;
+ use crate::writer::datum::GenericDatumWriter;
+ use crate::{AvroResult, Decimal};
+
+ #[track_caller]
+ fn assert_roundtrip<T>(value: T, schema: &Schema, schemata: Vec<&Schema>) -> AvroResult<()>
+ where
+ T: Serialize + DeserializeOwned + PartialEq + Debug + Clone,
+ {
+ let buf = GenericDatumWriter::builder(schema)
+ .schemata(schemata.clone())?
+ .build()?
+ .write_ser_to_vec(&value)?;
+
+ let decoded_value: T = GenericDatumReader::builder(schema)
+ .writer_schemata(schemata)?
+ .build()?
+ .read_deser(&mut Cursor::new(buf))?;
+
+ assert_eq!(decoded_value, value);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3955_decode_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
+ pub enum SourceType {
+ Sozu,
+ Haproxy,
+ HaproxyTcp,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct AccessLog {
+ source: SourceType,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "AccessLog",
+ "namespace": "com.clevercloud.accesslogs.common.avro",
+ "type": "record",
+ "fields": [{
+ "name": "source",
+ "type": {
+ "type": "enum",
+ "name": "SourceType",
+ "items": "string",
+ "symbols": ["SOZU", "HAPROXY", "HAPROXY_TCP"]
+ }
+ }]
+ }"#,
+ )?;
+
+ let data = AccessLog {
+ source: SourceType::Sozu,
+ };
+
+ assert_roundtrip(data, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_decode_enum_invalid_data() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
+ pub enum SourceType {
+ Sozu,
+ Haproxy,
+ HaproxyTcp,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct AccessLog {
+ source: SourceType,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "AccessLog",
+ "namespace": "com.clevercloud.accesslogs.common.avro",
+ "type": "record",
+ "fields": [{
+ "name": "source",
+ "type": {
+ "type": "enum",
+ "name": "SourceType",
+ "items": "string",
+ "symbols": ["SOZU", "HAPROXY", "HAPROXY_TCP"]
+ }
+ }]
+ }"#,
+ )?;
+
+ // Contains index 3 (4th symbol)
+ let data_with_unknown_index = vec![6u8];
+
+ let error = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser::<AccessLog>(&mut Cursor::new(data_with_unknown_index))
+ .unwrap_err();
+
+ assert_eq!(error.to_string(), "Enum symbol index out of bounds: 3");
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_nested_struct() -> TestResult {
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
+ struct Test {
+ a: i64,
+ b: String,
+ c: Decimal,
+ }
+
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq, Eq)]
+ struct TestInner {
+ a: Test,
+ b: i32,
+ }
+
+ let schemas = Schema::parse_list([
+ r#"{
+ "name": "Test",
+ "type": "record",
+ "fields": [
+ {
+ "name": "a",
+ "type": "long"
+ },
+ {
+ "name": "b",
+ "type": "string"
+ },
+ {
+ "name": "c",
+ "type": {
+ "type": "bytes",
+ "logicalType": "decimal",
+ "precision": 4,
+ "scale": 2
+ }
+ }
+ ]
+ }"#,
+ r#"{
+ "name": "TestInner",
+ "type": "record",
+ "fields": [
+ {
+ "name": "a",
+ "type": "Test"
+ },
+ {
+ "name": "b",
+ "type": "int"
+ }
+ ]
+ }"#,
+ ])?;
+
+ let test = Test {
+ a: 27,
+ b: "foo".to_string(),
+ c: Decimal::from(vec![1, 24]),
+ };
+
+ assert_roundtrip(test.clone(), &schemas[0], Vec::new())?;
+
+ let test_inner = TestInner { a: test, b: 35 };
+
+ assert_roundtrip(test_inner, &schemas[1], vec![&schemas[0]])?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_external_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ pub enum UnitExternalEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitExternalEnum {
+ a: UnitExternalEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "enum",
+ "name": "UnitExternalEnum",
+ "items": "string",
+ "symbols": ["Val1", "Val2"]
+ }
+ }]
+ }"#,
+ )?;
+
+ let alt_schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": []
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitExternalEnum {
+ a: UnitExternalEnum::Val1,
+ };
+ assert_roundtrip(value.clone(), &schema, Vec::new())?;
+ assert_roundtrip(value, &alt_schema, Vec::new())?;
+
+ let value = TestUnitExternalEnum {
+ a: UnitExternalEnum::Val2,
+ };
+ assert_roundtrip(value.clone(), &alt_schema, Vec::new())?;
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_internal_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(tag = "t")]
+ pub enum UnitInternalEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitInternalEnum {
+ a: UnitInternalEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitInternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "record",
+ "name": "UnitInternalEnum",
+ "fields": [{
+ "name": "t",
+ "type": "string"
+ }]
+ }
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitInternalEnum {
+ a: UnitInternalEnum::Val1,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ let value = TestUnitInternalEnum {
+ a: UnitInternalEnum::Val2,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_adjacent_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(tag = "t", content = "v")]
+ pub enum UnitAdjacentEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitAdjacentEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": {
+ "type": "record",
+ "name": "UnitAdjacentEnum",
+ "fields": [
+ {
+ "name": "t",
+ "type": {
+ "type": "enum",
+ "name": "t",
+ "symbols": ["Val1", "Val2"]
+ }
+ },
+ {
+ "name": "v",
+ "default": null,
+ "type": ["null"]
+ }
+ ]
+ }
+ }]
+ }"#,
+ )?;
+
+ let value = TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum::Val1,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ let value = TestUnitAdjacentEnum {
+ a: UnitAdjacentEnum::Val2,
+ };
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_untagged_unit_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ #[serde(untagged)]
+ pub enum UnitUntaggedEnum {
+ Val1,
+ Val2,
+ }
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestUnitUntaggedEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": ["null"]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum::Val1,
+ };
+ assert_roundtrip(value1.clone(), &schema, Vec::new())?;
+
+ let value2 = TestUnitUntaggedEnum {
+ a: UnitUntaggedEnum::Val2,
+ };
+ let buf = GenericDatumWriter::builder(&schema)
+ .build()?
+ .write_ser_to_vec(&value1)?;
+
+ let decoded_value: TestUnitUntaggedEnum = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser(&mut Cursor::new(buf))?;
+
+ // Val2 cannot troundtrip. All unit variants are serialized to the same null.
+ // This also doesn't roundtrip in serde_json.
+ assert_ne!(value2, decoded_value);
+ assert_eq!(decoded_value, value1);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_mixed_enum() -> TestResult {
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq)]
+ struct TestNullExternalEnum {
+ a: NullExternalEnum,
+ }
+
+ #[derive(Debug, Deserialize, Serialize, Clone, PartialEq)]
+ enum NullExternalEnum {
+ Val1,
+ Val2(),
+ Val3(()),
+ Val4(u64),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestNullExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": []
+ },
+ {
+ "name": "Val3",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "null"
+ }]
+ },
+ {
+ "name": "Val4",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": {
+ "type": "fixed",
+ "name": "u64",
+ "size": 8
+ }
+ }]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let data = [
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val1,
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val2(),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val2(),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val3(()),
+ },
+ TestNullExternalEnum {
+ a: NullExternalEnum::Val4(123),
+ },
+ ];
+
+ for value in data {
+ assert_roundtrip(value, &schema, Vec::new())?;
+ }
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_single_value_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum SingleValueExternalEnum {
+ Double(f64),
+ String(String),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestSingleValueExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Double",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "double"
+ }]
+ },
+ {
+ "name": "String",
+ "type": "record",
+ "org.apache.avro.rust.union_of_records": true,
+ "fields": [{
+ "name": "field_0",
+ "type": "string"
+ }]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let alt_schema = Schema::parse_str(
+ r#"{
+ "name": "TestSingleValueExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": ["double", "string"]
+ }]
+ }"#,
+ )?;
+
+ let double = TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum::Double(64.0),
+ };
+ assert_roundtrip(double.clone(), &schema, Vec::new())?;
+ assert_roundtrip(double, &alt_schema, Vec::new())?;
+
+ let string = TestSingleValueExternalEnum {
+ a: SingleValueExternalEnum::String("test".to_string()),
+ };
+ assert_roundtrip(string.clone(), &schema, Vec::new())?;
+ assert_roundtrip(string, &alt_schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_struct_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestStructExternalEnum {
+ a: StructExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum StructExternalEnum {
+ Val1 { x: f32, y: f32 },
+ Val2 { x: f32, y: f32 },
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestStructExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "x",
+ "type": "float"
+ },
+ {
+ "name": "y",
+ "type": "float"
+ }
+ ]
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": [
+ {
+ "name": "x",
+ "type": "float"
+ },
+ {
+ "name": "y",
+ "type": "float"
+ }
+ ]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestStructExternalEnum {
+ a: StructExternalEnum::Val1 { x: 1.0, y: 2.0 },
+ };
+
+ assert_roundtrip(value1, &schema, Vec::new())?;
+
+ let value2 = TestStructExternalEnum {
+ a: StructExternalEnum::Val2 { x: 2.0, y: 1.0 },
+ };
+
+ assert_roundtrip(value2, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_struct_flatten() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct S1 {
+ f1: String,
+ #[serde(flatten)]
+ inner: S2,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct S2 {
+ f2: String,
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "S1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "f1",
+ "type": "string"
+ },
+ {
+ "name": "f2",
+ "type": "string"
+ }
+ ]
+ }"#,
+ )?;
+
+ let value = S1 {
+ f1: "Hello".to_owned(),
+ inner: S2 {
+ f2: "World".to_owned(),
+ },
+ };
+
+ assert_roundtrip(value, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_tuple_enum() -> TestResult {
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ struct TestTupleExternalEnum {
+ a: TupleExternalEnum,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+ enum TupleExternalEnum {
+ Val1(f32, f32),
+ Val2(f32, f32, f32),
+ }
+
+ let schema = Schema::parse_str(
+ r#"{
+ "name": "TestTupleExternalEnum",
+ "type": "record",
+ "fields": [{
+ "name": "a",
+ "type": [
+ {
+ "name": "Val1",
+ "type": "record",
+ "fields": [
+ {
+ "name": "field_0",
+ "type": "float"
+ },
+ {
+ "name": "field_1",
+ "type": "float"
+ }
+ ]
+ },
+ {
+ "name": "Val2",
+ "type": "record",
+ "fields": [
+ {
+ "name": "field_0",
+ "type": "float"
+ },
+ {
+ "name": "field_1",
+ "type": "float"
+ },
+ {
+ "name": "field_2",
+ "type": "float"
+ }
+ ]
+ }
+ ]
+ }]
+ }"#,
+ )?;
+
+ let value1 = TestTupleExternalEnum {
+ a: TupleExternalEnum::Val1(1.0, 2.0),
+ };
+
+ assert_roundtrip(value1, &schema, Vec::new())?;
+
+ let value2 = TestTupleExternalEnum {
+ a: TupleExternalEnum::Val1(2.0, 1.0),
+ };
+
+ assert_roundtrip(value2, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_date() -> TestResult {
+ let schema = Schema::Date;
+ assert_roundtrip(1i32, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_time_millis() -> TestResult {
+ let schema = Schema::TimeMillis;
+ assert_roundtrip(1i32, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_time_micros() -> TestResult {
+ let schema = Schema::TimeMicros;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_timestamp_millis() -> TestResult {
+ let schema = Schema::TimestampMillis;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_timestamp_micros() -> TestResult {
+ let schema = Schema::TimestampMicros;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3916_timestamp_nanos() -> TestResult {
+ let schema = Schema::TimestampNanos;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3853_local_timestamp_millis() -> TestResult {
+ let schema = Schema::LocalTimestampMillis;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3853_local_timestamp_micros() -> TestResult {
+ let schema = Schema::LocalTimestampMicros;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_3916_local_timestamp_nanos() -> TestResult {
+ let schema = Schema::LocalTimestampNanos;
+ assert_roundtrip(1i64, &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_uuid() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "fixed",
+ "logicalType": "uuid",
+ "size": 16,
+ "name": "uuid"
+ }"#,
+ )?;
+
+ let alt_schema = Schema::Uuid(UuidSchema::String);
+
+ let uuid = Uuid::parse_str("9ec535ff-3e2a-45bd-91d3-0a01321b5a49")?;
+
+ assert_roundtrip(uuid, &schema, Vec::new())?;
+
+ let buf = GenericDatumWriter::builder(&alt_schema)
+ // .human_readable(true)
+ .build()?
+ .write_ser_to_vec(&uuid)?;
+
+ let decoded_value: Uuid = GenericDatumReader::builder(&alt_schema)
+ .human_readable(true)
+ .build()?
+ .read_deser(&mut Cursor::new(buf))?;
+
+ assert_eq!(decoded_value, uuid);
+
+ Ok(())
+ }
+
+ #[derive(Debug)]
+ struct Bytes(Vec<u8>);
+
+ impl<'de> Deserialize<'de> for Bytes {
+ fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+ where
+ D: serde::Deserializer<'de>,
+ {
+ struct BytesVisitor;
+ impl Visitor<'_> for BytesVisitor {
+ type Value = Bytes;
+
+ fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
+ formatter.write_str("a byte array")
+ }
+
+ fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
+ where
+ E: serde::de::Error,
+ {
+ Ok(Bytes(v.to_vec()))
+ }
+ }
+ deserializer.deserialize_bytes(BytesVisitor)
+ }
+ }
+
+ #[test]
+ fn avro_3892_deserialize_bytes_from_decimal() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "bytes",
+ "logicalType": "decimal",
+ "precision": 4,
+ "scale": 2
+ }"#,
+ )?;
+ let schema_union = Schema::parse_str(
+ r#"[
+ "null",
+ {
+ "type": "bytes",
+ "logicalType": "decimal",
+ "precision": 4,
+ "scale": 2
+ }
+ ]"#,
+ )?;
+
+ let expected_bytes = BigInt::from(123456789).to_signed_bytes_be();
+ let value = Decimal::from(&expected_bytes);
+ let buf = GenericDatumWriter::builder(&schema)
+ .build()?
+ .write_ser_to_vec(&value)?;
+
+ let decoded_value: Bytes = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser(&mut Cursor::new(buf))?;
+
+ assert_eq!(decoded_value.0, expected_bytes);
+
+ let buf = GenericDatumWriter::builder(&schema_union)
+ .build()?
+ .write_ser_to_vec(&Some(value))?;
+
+ let decoded_value: Option<Bytes> = GenericDatumReader::builder(&schema_union)
+ .build()?
+ .read_deser(&mut Cursor::new(buf))?;
+
+ assert_eq!(decoded_value.unwrap().0, expected_bytes);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_414_deserialize_char_from_string() -> TestResult {
+ let schema = Schema::String;
+
+ assert_roundtrip('a', &schema, Vec::new())?;
+ assert_roundtrip('👹', &schema, Vec::new())?;
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_414_deserialize_char_from_long_string() -> TestResult {
+ let schema = Schema::String;
+ let buf = GenericDatumWriter::builder(&schema)
+ .build()?
+ .write_ser_to_vec(&"avro")?;
+
+ let error = GenericDatumReader::builder(&schema)
+ .build()?
+ .read_deser::<char>(&mut Cursor::new(buf))
+ .unwrap_err();
+
+ assert_eq!(
+ error.to_string(),
+ r#"Failed to deserialize value of type char using schema String: Read more than one character: "avro""#
+ );
+
+ Ok(())
+ }
+}
(avro-rs) 05/05: temp
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a commit to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
commit c3e182567cd9251df88c44bbf9bf2cd967725ae5
Author: default <ad...@kriskras99.nl>
AuthorDate: Thu Mar 19 12:47:49 2026 +0000
temp
---
avro/src/bigdecimal.rs | 45 +-
avro/src/error.rs | 15 +-
avro/src/schema/mod.rs | 64 +
avro/src/serde/derive.rs | 384 +++++-
avro/src/serde/mod.rs | 5 +-
avro/src/serde/ser_schema/mod.rs | 1723 ++++++++++---------------
avro/src/serde/ser_schema/record/mod.rs | 58 +-
avro/src/serde/with.rs | 309 +++++
avro/src/writer/mod.rs | 2 +-
avro/tests/serde_human_readable_false.rs | 2 +-
avro/tests/serde_human_readable_true.rs | 4 +-
avro_derive/tests/derive.proptest-regressions | 8 +
avro_derive/tests/derive.rs | 27 +-
avro_derive/tests/serde.rs | 9 +-
14 files changed, 1516 insertions(+), 1139 deletions(-)
diff --git a/avro/src/bigdecimal.rs b/avro/src/bigdecimal.rs
index 7a6a379..ee26ee1 100644
--- a/avro/src/bigdecimal.rs
+++ b/avro/src/bigdecimal.rs
@@ -190,6 +190,47 @@ mod tests {
big_decimal: BigDecimal,
}
+ let schema_str = r#"
+ {
+ "type": "record",
+ "name": "Test",
+ "fields": [
+ {
+ "name": "big_decimal",
+ "type": "string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema_str)?;
+
+ let test = Test::default();
+
+ // write a record
+ let mut writer = Writer::new(&schema, Vec::new())?;
+ writer.append_ser(test.clone())?;
+
+ let wrote_data = writer.into_inner()?;
+
+ // read record
+ let mut reader = Reader::new(&wrote_data[..])?;
+
+ let value = reader.next().unwrap()?;
+
+ assert_eq!(test, from_value::<Test>(&value)?);
+
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_338_deserialize_serde_way_with_bigdecimal() -> TestResult {
+ #[derive(Clone, PartialEq, Eq, Debug, Default, serde::Deserialize, serde::Serialize)]
+ #[serde(rename = "test")]
+ struct Test {
+ #[serde(with = "crate::serde::bigdecimal")]
+ big_decimal: BigDecimal,
+ }
+
let schema_str = r#"
{
"type": "record",
@@ -216,11 +257,11 @@ mod tests {
let wrote_data = writer.into_inner()?;
// read record
- let mut reader = Reader::new(&wrote_data[..])?;
+ let mut reader = Reader::new(&wrote_data[..])?.into_deser_iter();
let value = reader.next().unwrap()?;
- assert_eq!(test, from_value::<Test>(&value)?);
+ assert_eq!(test, value);
Ok(())
}
diff --git a/avro/src/error.rs b/avro/src/error.rs
index 85c7fd8..e4c09cd 100644
--- a/avro/src/error.rs
+++ b/avro/src/error.rs
@@ -559,7 +559,7 @@ pub enum Details {
#[error("Failed to serialize value into Avro value: {0}")]
SerializeValue(String),
- #[error("Failed to serialize value of type {value_type} using schema {schema:?}: {value}")]
+ #[error("Failed to serialize value of type `{value_type}` using Schema::{schema:?}: {value}")]
SerializeValueWithSchema {
value_type: &'static str,
value: String,
@@ -572,15 +572,18 @@ pub enum Details {
schema: RecordSchema,
},
- #[error("Failed to serialize field '{field_name}' for record {record_schema:?}: {error}")]
+ #[error("Failed to serialize field '{field_name}' of record {record_schema:?}: {error}")]
SerializeRecordFieldWithSchema {
field_name: String,
- record_schema: Schema,
- error: Box<Error>,
+ record_schema: RecordSchema,
+ error: String,
},
- #[error("Missing default for skipped field '{field_name}' for schema {schema:?}")]
- MissingDefaultForSkippedField { field_name: String, schema: Schema },
+ #[error("Missing default for skipped field '{field_name}' of schema {schema:?}")]
+ MissingDefaultForSkippedField {
+ field_name: String,
+ schema: RecordSchema,
+ },
#[error("Failed to deserialize Avro value into value: {0}")]
DeserializeValue(String),
diff --git a/avro/src/schema/mod.rs b/avro/src/schema/mod.rs
index 40476cb..ccaa9f8 100644
--- a/avro/src/schema/mod.rs
+++ b/avro/src/schema/mod.rs
@@ -45,6 +45,7 @@ use serde::{
ser::{SerializeMap, SerializeSeq},
};
use serde_json::{Map, Value as JsonValue};
+use std::borrow::Cow;
use std::fmt::Formatter;
use std::{
collections::{BTreeMap, HashMap, HashSet},
@@ -784,6 +785,69 @@ impl Schema {
}
Ok(())
}
+
+ /// Derive a name for this schema.
+ ///
+ /// The name is a valid schema name and will be unique if the named
+ /// schemas in this schema have unique names.
+ pub(crate) fn unique_normalized_name(&self) -> Cow<'static, str> {
+ match self {
+ Schema::Null => Cow::Borrowed("n"),
+ Schema::Boolean => Cow::Borrowed("B"),
+ Schema::Int => Cow::Borrowed("i"),
+ Schema::Long => Cow::Borrowed("l"),
+ Schema::Float => Cow::Borrowed("f"),
+ Schema::Double => Cow::Borrowed("d"),
+ Schema::Bytes => Cow::Borrowed("b"),
+ Schema::String => Cow::Borrowed("s"),
+ Schema::Array(array) => {
+ Cow::Owned(format!("a_{}", array.items.unique_normalized_name()))
+ }
+ Schema::Map(map) => Cow::Owned(format!("m_{}", map.types.unique_normalized_name())),
+ Schema::Union(union) => {
+ let mut name = format!("u{}", union.schemas.len());
+ for schema in &union.schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ Cow::Owned(name)
+ }
+ Schema::BigDecimal => Cow::Borrowed("bd"),
+ Schema::Date => Cow::Borrowed("D"),
+ Schema::TimeMillis => Cow::Borrowed("t"),
+ Schema::TimeMicros => Cow::Borrowed("tm"),
+ Schema::TimestampMillis => Cow::Borrowed("T"),
+ Schema::TimestampMicros => Cow::Borrowed("TM"),
+ Schema::TimestampNanos => Cow::Borrowed("TN"),
+ Schema::LocalTimestampMillis => Cow::Borrowed("L"),
+ Schema::LocalTimestampMicros => Cow::Borrowed("LM"),
+ Schema::LocalTimestampNanos => Cow::Borrowed("LN"),
+ Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Bytes,
+ precision,
+ scale,
+ }) => Cow::Owned(format!("db_{precision}_{scale}")),
+ Schema::Uuid(UuidSchema::Bytes) => Cow::Borrowed("ub"),
+ Schema::Uuid(UuidSchema::String) => Cow::Borrowed("us"),
+ Schema::Record(RecordSchema { name, .. })
+ | Schema::Enum(EnumSchema { name, .. })
+ | Schema::Fixed(FixedSchema { name, .. })
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(FixedSchema { name, .. }),
+ ..
+ })
+ | Schema::Uuid(UuidSchema::Fixed(FixedSchema { name, .. }))
+ | Schema::Duration(FixedSchema { name, .. })
+ | Schema::Ref { name } => {
+ let name: String = name
+ .to_string()
+ .chars()
+ .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
+ .collect();
+ Cow::Owned(format!("r{}_{}", name.len(), name))
+ }
+ }
+ }
}
impl Serialize for Schema {
diff --git a/avro/src/serde/derive.rs b/avro/src/serde/derive.rs
index 33c4f5b..255da4b 100644
--- a/avro/src/serde/derive.rs
+++ b/avro/src/serde/derive.rs
@@ -558,27 +558,6 @@ macro_rules! impl_array_schema (
impl_array_schema!([T] where T: AvroSchemaComponent);
impl_array_schema!(Vec<T> where T: AvroSchemaComponent);
-// This doesn't work as the macro doesn't allow specifying the N parameter
-// impl_array_schema!([T; N] where T: AvroSchemaComponent);
-
-impl<const N: usize, T> AvroSchemaComponent for [T; N]
-where
- T: AvroSchemaComponent,
-{
- fn get_schema_in_ctxt(
- named_schemas: &mut HashSet<Name>,
- enclosing_namespace: NamespaceRef,
- ) -> Schema {
- Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build()
- }
-
- fn get_record_fields_in_ctxt(
- _: &mut HashSet<Name>,
- _: NamespaceRef,
- ) -> Option<Vec<RecordField>> {
- None
- }
-}
impl<T> AvroSchemaComponent for HashMap<String, T>
where
@@ -630,27 +609,35 @@ where
}
impl AvroSchemaComponent for core::time::Duration {
- /// The schema is [`Schema::Duration`] with the name `duration`.
+ /// The schema is [`Schema::Record`] with the name `Duration`.
///
- /// This is a lossy conversion as this Avro type does not store the amount of nanoseconds.
+ /// It has two fields:
+ /// - `secs` with the schema `Schema::Fixed(name: "u64", size: 8)`
+ /// - `nanos` with the schema `Schema::Long`
fn get_schema_in_ctxt(
named_schemas: &mut HashSet<Name>,
enclosing_namespace: NamespaceRef,
) -> Schema {
- let name = Name::new_with_enclosing_namespace("duration", enclosing_namespace)
+ let name = Name::new_with_enclosing_namespace("Duration", enclosing_namespace)
.expect("Name is valid");
if named_schemas.contains(&name) {
Schema::Ref { name }
} else {
- let schema = Schema::Duration(FixedSchema {
- name: name.clone(),
- aliases: None,
- doc: None,
- size: 12,
- attributes: Default::default(),
- });
- named_schemas.insert(name);
- schema
+ named_schemas.insert(name.clone());
+ Schema::record(name)
+ .fields(vec![
+ // Secs is an u64
+ RecordField::builder()
+ .name("secs")
+ .schema(u64::get_schema_in_ctxt(named_schemas, enclosing_namespace))
+ .build(),
+ // Nanos is an u32
+ RecordField::builder()
+ .name("nanos")
+ .schema(Schema::Long)
+ .build(),
+ ])
+ .build()
}
}
@@ -666,6 +653,8 @@ impl AvroSchemaComponent for uuid::Uuid {
/// The schema is [`Schema::Uuid`] with the name `uuid`.
///
/// The underlying schema is [`Schema::Fixed`] with a size of 16.
+ ///
+ /// If you're using `human_readable: true` you need to override this schema with a `Schema::String`.
fn get_schema_in_ctxt(
named_schemas: &mut HashSet<Name>,
enclosing_namespace: NamespaceRef,
@@ -788,13 +777,197 @@ impl AvroSchemaComponent for i128 {
}
}
+/// Schema definition for `[T; N]`
+///
+/// Schema is defined as follows:
+/// - 0-sized arrays: [`Schema::Null`]
+/// - 1-sized arrays: `T::get_schema_in_ctxt`
+/// - N-sized arrays: [`Schema::Record`] with a field for every index
+///
+/// If you need or want a [`Schema::Array`], [`Schema::Bytes`], or [`Schema::Fixed`] instead,
+/// use [`apache_avro::serde::array`], [`apache_avro::serde::bytes`], or [`apache_avro::serde::fixed`] respectively.
+///
+/// [`apache_avro::serde::array`]: crate::serde::array
+/// [`apache_avro::serde::bytes`]: crate::serde::bytes
+/// [`apache_avro::serde::fixed`]: crate::serde::fixed
+impl<const N: usize, T: AvroSchemaComponent> AvroSchemaComponent for [T; N] {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ if N == 0 {
+ Schema::Null
+ } else if N == 1 {
+ T::get_schema_in_ctxt(named_schemas, enclosing_namespace)
+ } else {
+ let t_schema = T::get_schema_in_ctxt(named_schemas, enclosing_namespace);
+ let name = Name::new_with_enclosing_namespace(
+ format!("A{N}_{}", t_schema.unique_normalized_name()),
+ enclosing_namespace,
+ )
+ .expect("Name is valid");
+ if named_schemas.contains(&name) {
+ Schema::Ref { name }
+ } else {
+ named_schemas.insert(name.clone());
+
+ let t_default = T::field_default();
+ // If T is a named schema or contains named schemas, they'll now be a reference.
+ let t_ref = T::get_schema_in_ctxt(named_schemas, enclosing_namespace);
+ let fields = std::iter::once(
+ RecordField::builder()
+ .name("field_0".to_string())
+ .schema(t_schema)
+ .maybe_default(t_default.clone())
+ .build(),
+ )
+ .chain((1..N).map(|n| {
+ RecordField::builder()
+ .name(format!("field_{n}"))
+ .schema(t_ref.clone())
+ .maybe_default(t_default.clone())
+ .build()
+ }))
+ .collect();
+
+ Schema::record(name).fields(fields).build()
+ }
+ }
+ }
+
+ fn get_record_fields_in_ctxt(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ if N == 0 {
+ None
+ } else if N == 1 {
+ T::get_record_fields_in_ctxt(named_schemas, enclosing_namespace)
+ } else {
+ let t_schema = T::get_schema_in_ctxt(named_schemas, enclosing_namespace);
+ let t_default = T::field_default();
+ // If T is a named schema or contains named schemas, they'll now be a reference.
+ let t_ref = T::get_schema_in_ctxt(named_schemas, enclosing_namespace);
+ let fields = std::iter::once(
+ RecordField::builder()
+ .name("field_0".to_string())
+ .schema(t_schema)
+ .maybe_default(t_default.clone())
+ .build(),
+ )
+ .chain((1..N).map(|n| {
+ RecordField::builder()
+ .name(format!("field_{n}"))
+ .schema(t_ref.clone())
+ .maybe_default(t_default.clone())
+ .build()
+ }))
+ .collect();
+ Some(fields)
+ }
+ }
+
+ /// `None` for 0-sized and N-sized arrays, `T::field_default` for 1-sized arrays
+ fn field_default() -> Option<serde_json::Value> {
+ if N == 1 { T::field_default() } else { None }
+ }
+}
+
+/// Schema definition for `(T₁, T₂, …, Tₙ)`.
+///
+/// Implemented for tuples of up to 16 elements.
+///
+/// Schema is defined as follows:
+/// - 1-tuple: `T::get_schema_in_ctxt`
+/// - N-tuple: [`Schema::Record`] with a field for every element
+#[cfg_attr(docsrs, doc(fake_variadic))]
+impl<T: AvroSchemaComponent> AvroSchemaComponent for (T,) {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ T::get_schema_in_ctxt(named_schemas, enclosing_namespace)
+ }
+
+ fn get_record_fields_in_ctxt(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ T::get_record_fields_in_ctxt(named_schemas, enclosing_namespace)
+ }
+
+ /// `None` for N-tuples, `T::field_default()` for 1-tuple.
+ fn field_default() -> Option<serde_json::Value> {
+ T::field_default()
+ }
+}
+
+macro_rules! tuple_impls {
+ ($($len:expr => ($($name:ident)+))+) => {
+ $(
+ #[cfg_attr(docsrs, doc(hidden))]
+ impl<$($name: AvroSchemaComponent),+> AvroSchemaComponent for ($($name),+) {
+ fn get_schema_in_ctxt(named_schemas: &mut HashSet<Name>, enclosing_namespace: NamespaceRef) -> Schema {
+ let schemas: [Schema; $len] = [$($name::get_schema_in_ctxt(named_schemas, enclosing_namespace)),+];
+
+ let mut name = format!("T{}", $len);
+ for schema in &schemas {
+ name.push('_');
+ name.push_str(&schema.unique_normalized_name());
+ }
+ let name = Name::new_with_enclosing_namespace(name, enclosing_namespace).expect("Name is valid");
+
+ if named_schemas.contains(&name) {
+ Schema::Ref { name }
+ } else {
+ named_schemas.insert(name.clone());
+
+ let defaults: [Option<serde_json::Value>; $len] = [$($name::field_default()),+];
+
+ let fields = schemas.into_iter().zip(defaults.into_iter()).enumerate().map(|(n, (schema, default))| {
+ RecordField::builder()
+ .name(format!("field_{n}"))
+ .schema(schema)
+ .maybe_default(default)
+ .build()
+ }).collect();
+
+ Schema::record(name).fields(fields).build()
+ }
+ }
+ }
+ )+
+ }
+}
+
+tuple_impls! {
+ 2 => (T0 T1)
+ 3 => (T0 T1 T2)
+ 4 => (T0 T1 T2 T3)
+ 5 => (T0 T1 T2 T3 T4)
+ 6 => (T0 T1 T2 T3 T4 T5)
+ 7 => (T0 T1 T2 T3 T4 T5 T6)
+ 8 => (T0 T1 T2 T3 T4 T5 T6 T7)
+ 9 => (T0 T1 T2 T3 T4 T5 T6 T7 T8)
+ 10 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9)
+ 11 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10)
+ 12 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11)
+ 13 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12)
+ 14 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13)
+ 15 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14)
+ 16 => (T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15)
+}
+
#[cfg(test)]
mod tests {
+ use crate::reader::datum::GenericDatumReader;
+ use crate::writer::datum::GenericDatumWriter;
use crate::{
AvroSchema, Schema,
schema::{FixedSchema, Name},
};
use apache_avro_test_helper::TestResult;
+ use std::io::Cursor;
#[test]
fn avro_rs_401_str() -> TestResult {
@@ -823,23 +996,12 @@ mod tests {
Ok(())
}
- #[test]
- fn avro_rs_401_array() -> TestResult {
- let schema = <[u8; 55]>::get_schema();
- assert_eq!(schema, Schema::array(Schema::Int).build());
-
- Ok(())
- }
-
#[test]
fn avro_rs_401_option_ref_slice_array() -> TestResult {
- let schema = <Option<&[[u8; 55]]>>::get_schema();
+ let schema = <Option<&[u8]>>::get_schema();
assert_eq!(
schema,
- Schema::union(vec![
- Schema::Null,
- Schema::array(Schema::array(Schema::Int).build()).build()
- ])?
+ Schema::union(vec![Schema::Null, Schema::array(Schema::Int).build()])?
);
Ok(())
@@ -927,4 +1089,134 @@ mod tests {
fn avro_rs_489_option_option() {
<Option<Option<i32>>>::get_schema();
}
+
+ #[test]
+ fn avro_rs_xxx_std_time_duration() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "record",
+ "name": "Duration",
+ "fields": [
+ { "name": "secs", "type": {"type": "fixed", "name": "u64", "size": 8} },
+ { "name": "nanos", "type": "long" }
+ ]
+ }"#,
+ )?;
+ let zero = std::time::Duration::ZERO;
+ let max = std::time::Duration::MAX;
+ assert_eq!(schema, std::time::Duration::get_schema());
+
+ let writer = GenericDatumWriter::builder(&schema).build()?;
+ let written_zero = writer.write_ser_to_vec(&zero)?;
+ let written_max = writer.write_ser_to_vec(&max)?;
+
+ let reader = GenericDatumReader::builder(&schema).build()?;
+ let read_zero = reader.read_deser(&mut Cursor::new(written_zero))?;
+ assert_eq!(zero, read_zero);
+ let read_max = reader.read_deser(&mut Cursor::new(written_max))?;
+ assert_eq!(max, read_max);
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_0_array() -> TestResult {
+ assert_eq!(Schema::Null, <[String; 0]>::get_schema());
+ assert_eq!(Schema::Null, <[(); 0]>::get_schema());
+ assert_eq!(Schema::Null, <[bool; 0]>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_1_array() -> TestResult {
+ assert_eq!(Schema::String, <[String; 1]>::get_schema());
+ assert_eq!(Schema::Null, <[(); 1]>::get_schema());
+ assert_eq!(Schema::Boolean, <[bool; 1]>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_n_array() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "record",
+ "name": "A5_s",
+ "fields": [
+ { "name": "field_0", "type": "string" },
+ { "name": "field_1", "type": "string" },
+ { "name": "field_2", "type": "string" },
+ { "name": "field_3", "type": "string" },
+ { "name": "field_4", "type": "string" }
+ ]
+ }"#,
+ )?;
+
+ assert_eq!(schema, <[String; 5]>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_n_array_complex_type() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "record",
+ "name": "A2_u2_n_r4_uuid",
+ "fields": [
+ { "name": "field_0", "type": ["null", {"type": "fixed", "logicalType": "uuid", "size": 16, "name": "uuid"}], "default": null },
+ { "name": "field_1", "type": ["null", "uuid"], "default": null }
+ ]
+ }"#,
+ )?;
+
+ assert_eq!(schema, <[Option<uuid::Uuid>; 2]>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_1_tuple() -> TestResult {
+ assert_eq!(Schema::String, <(String,)>::get_schema());
+ assert_eq!(Schema::Null, <((),)>::get_schema());
+ assert_eq!(Schema::Boolean, <(bool,)>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rs_xxx_n_tuple() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "record",
+ "name": "T5_s_i_l_B_n",
+ "fields": [
+ { "name": "field_0", "type": "string" },
+ { "name": "field_1", "type": "int" },
+ { "name": "field_2", "type": "long" },
+ { "name": "field_3", "type": "boolean" },
+ { "name": "field_4", "type": "null" }
+ ]
+ }"#,
+ )?;
+
+ assert_eq!(schema, <(String, i32, i64, bool, ())>::get_schema());
+ Ok(())
+ }
+
+ #[test]
+ fn avro_rx_xxx_n_tuple_complex_type() -> TestResult {
+ let schema = Schema::parse_str(
+ r#"{
+ "type": "record",
+ "name": "T3_u2_n_r4_uuid_r4_uuid_s",
+ "fields": [
+ { "name": "field_0", "type": ["null", {"type": "fixed", "logicalType": "uuid", "size": 16, "name": "uuid"}], "default": null },
+ { "name": "field_1", "type": "uuid" },
+ { "name": "field_2", "type": "string" }
+ ]
+ }"#,
+ )?;
+
+ assert_eq!(
+ schema,
+ <(Option<uuid::Uuid>, uuid::Uuid, String)>::get_schema()
+ );
+ Ok(())
+ }
}
diff --git a/avro/src/serde/mod.rs b/avro/src/serde/mod.rs
index 256963d..3b1718b 100644
--- a/avro/src/serde/mod.rs
+++ b/avro/src/serde/mod.rs
@@ -118,7 +118,10 @@ mod with;
pub use de::from_value;
pub use derive::{AvroSchema, AvroSchemaComponent};
pub use ser::to_value;
-pub use with::{bytes, bytes_opt, fixed, fixed_opt, slice, slice_opt};
+pub use with::{
+ array, array_opt, bigdecimal, bigdecimal_opt, bytes, bytes_opt, fixed, fixed_opt, slice,
+ slice_opt,
+};
#[doc(hidden)]
pub use derive::get_record_fields_in_ctxt;
diff --git a/avro/src/serde/ser_schema/mod.rs b/avro/src/serde/ser_schema/mod.rs
index 829cec4..c971bff 100644
--- a/avro/src/serde/ser_schema/mod.rs
+++ b/avro/src/serde/ser_schema/mod.rs
@@ -268,7 +268,7 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for SchemaAwareSerializer<'
}
fn serialize_u32(self, v: u32) -> Result<Self::Ok, Self::Error> {
- self.checked_write_long("i64", i64::from(v))
+ self.checked_write_long("u32", i64::from(v))
}
fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
@@ -334,7 +334,7 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for SchemaAwareSerializer<'
Schema::Union(union) => {
UnionSerializer::new(self.writer, union, self.config).serialize_str(v)
}
- _ => Err(self.error("string", "Expected Schema::String ")),
+ _ => Err(self.error("str", "Expected Schema::String | Schema::Uuid(String)")),
}
}
@@ -406,36 +406,36 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for SchemaAwareSerializer<'
}
_ => Err(self.error(
"unit struct",
- format!("Expected Schema::Record(name: {name}, fields: [])"),
+ format!(r#"Expected Schema::Record(name: "{name}", fields: [])"#),
)),
}
}
fn serialize_unit_variant(
self,
- name: &'static str,
+ _name: &'static str,
variant_index: u32,
variant: &'static str,
) -> Result<Self::Ok, Self::Error> {
match self.schema {
Schema::Enum(enum_schema) => {
// Plain enum
- if name.as_ptr() == SERIALIZING_SCHEMA_DEFAULT.as_ptr() || enum_schema.symbols[variant_index as usize] == variant {
+ if variant.as_ptr() == SERIALIZING_SCHEMA_DEFAULT.as_ptr() || enum_schema.symbols[variant_index as usize] == variant {
zig_i32(variant_index as i32, &mut *self.writer)
} else {
- Err(self.error("unit variant", format!(r#"Expected symbol "{name}" at index {variant_index} in enum"#)))
+ Err(self.error("unit variant", format!(r#"Expected symbol "{variant}" at index {variant_index} in enum"#)))
}
}
Schema::Union(union) => match self.get_resolved_union_variant(union, variant_index)? {
// Bare union
Schema::Null => zig_i32(variant_index as i32, &mut *self.writer),
- Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == variant => {
// Union of records
zig_i32(variant_index as i32, &mut *self.writer)
}
- _ => Err(self.error("unit variant", format!("Expected Schema::Null | Schema::Record(name: {name}, fields: []) at index {variant_index} in the union"))),
+ _ => Err(self.error("unit variant", format!("Expected Schema::Null | Schema::Record(name: {variant}, fields: []) at index {variant_index} in the union"))),
}
- _ => Err(self.error("unit variant", format!("Expected Schema::Enum(symbols[{variant_index}] == {name}) | Schema::Union(variants[{variant_index}] == Schema::Null | Schema::Record(name: {name}, fields: []))"))),
+ _ => Err(self.error("unit variant", format!("Expected Schema::Enum(symbols[{variant_index}] == {variant}) | Schema::Union(variants[{variant_index}] == Schema::Null | Schema::Record(name: {variant}, fields: []))"))),
}
}
@@ -527,7 +527,10 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for SchemaAwareSerializer<'
None,
)),
// This error case can only happen for len > 1
- _ => Err(self.error("tuple", "Expected Schema::Record(fields.len() == {len}")),
+ _ => Err(self.error(
+ "tuple",
+ format!("Expected Schema::Record(fields.len() == {len})"),
+ )),
}
}
@@ -725,47 +728,94 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for MapOrRecordSerializer
mod tests {
use super::*;
use crate::schema::FixedSchema;
- use crate::{
- Days, Duration, Millis, Months, Reader, Writer, decimal::Decimal, error::Details,
- from_value, schema::ResolvedSchema,
- };
+ use crate::{Days, Duration, Millis, Months, decimal::Decimal, schema::ResolvedSchema};
use apache_avro_test_helper::TestResult;
use bigdecimal::BigDecimal;
use num_bigint::{BigInt, Sign};
+ use pretty_assertions::assert_eq;
use serde::{Deserialize, Serialize};
- use serde_bytes::{ByteArray, Bytes};
+ use serde_bytes::Bytes;
use std::{
collections::{BTreeMap, HashMap},
marker::PhantomData,
};
use uuid::Uuid;
+ #[track_caller]
+ fn assert_serialize_err<T: Serialize>(
+ t: T,
+ schema: &Schema,
+ names: &HashMap<Name, &Schema>,
+ expected: &str,
+ ) {
+ let config = Config {
+ names,
+ target_block_size: None,
+ human_readable: false,
+ };
+ let mut buffer = Vec::new();
+ let serializer = SchemaAwareSerializer::new(&mut buffer, schema, config).unwrap();
+ let error = t
+ .serialize(serializer)
+ .expect_err("This should not serialize");
+ assert_eq!(error.to_string(), expected);
+ }
+
+ #[track_caller]
+ fn assert_serialize<T: Serialize>(
+ t: T,
+ schema: &Schema,
+ names: &HashMap<Name, &Schema>,
+ expected: &[u8],
+ ) {
+ let config = Config {
+ names,
+ target_block_size: None,
+ human_readable: false,
+ };
+ let mut buffer = Vec::new();
+ let serializer = SchemaAwareSerializer::new(&mut buffer, schema, config).unwrap();
+ let bytes_written = t.serialize(serializer).expect("This should serialize");
+ assert_eq!(bytes_written, buffer.len());
+ assert_eq!(&buffer, expected);
+ }
+
#[test]
fn test_serialize_null() -> TestResult {
let schema = Schema::Null;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- None::<()>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- None::<i32>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- None::<String>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize((), &schema, &names, &[]);
+ assert_serialize_err(
+ None::<()>,
+ &schema,
+ &names,
+ "Failed to serialize value of type `none` using Schema::Null: Expected Schema::Union([Schema::Null, _])",
+ );
+ assert_serialize_err(
+ None::<i32>,
+ &schema,
+ &names,
+ "Failed to serialize value of type `none` using Schema::Null: Expected Schema::Union([Schema::Null, _])",
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ None::<String>,
+ &schema,
+ &names,
+ "Failed to serialize value of type `none` using Schema::Null: Expected Schema::Union([Schema::Null, _])",
+ );
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Null: Expected Schema::String | Schema::Uuid(String)",
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Null: Expected Schema::Union([Schema::Null, _])",
);
-
- assert_eq!(buffer.as_slice(), Vec::<u8>::new().as_slice());
Ok(())
}
@@ -773,59 +823,54 @@ mod tests {
#[test]
fn test_serialize_bool() -> TestResult {
let schema = Schema::Boolean;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- true.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- false.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize(true, &schema, &names, &[1]);
+ assert_serialize(false, &schema, &names, &[0]);
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Boolean: Expected Schema::String | Schema::Uuid(String)",
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Boolean: Expected Schema::Union([Schema::Null, _])",
);
- assert_eq!(buffer.as_slice(), &[1, 0]);
-
Ok(())
}
#[test]
fn test_serialize_int() -> TestResult {
let schema = Schema::Int;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 4u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 31u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 13u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 7i8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- (-57i16).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 129i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize(4u8, &schema, &names, &[8]);
+ assert_serialize(31u16, &schema, &names, &[62]);
+ assert_serialize(7i8, &schema, &names, &[14]);
+ assert_serialize(-57i16, &schema, &names, &[113]);
+ assert_serialize(129i32, &schema, &names, &[130, 2]);
+ assert_serialize_err(
+ 13u32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u32` using Schema::Int: Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Int: Expected Schema::String | Schema::Uuid(String)",
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Int: Expected Schema::Union([Schema::Null, _])",
);
-
- assert_eq!(buffer.as_slice(), &[8, 62, 26, 14, 113, 130, 2]);
Ok(())
}
@@ -833,35 +878,57 @@ mod tests {
#[test]
fn test_serialize_long() -> TestResult {
let schema = Schema::Long;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 4u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 31u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 13u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 291u64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 7i8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- (-57i16).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 129i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- (-432i64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize(13u32, &schema, &names, &[26]);
+ assert_serialize(-432i64, &schema, &names, &[223, 6]);
+ assert_serialize_err(
+ 4u8,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u8` using Schema::Long: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ 31u16,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u16` using Schema::Long: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
);
-
- assert_eq!(
- buffer.as_slice(),
- &[8, 62, 26, 198, 4, 14, 113, 130, 2, 223, 6]
+ assert_serialize_err(
+ 7i8,
+ &schema,
+ &names,
+ "Failed to serialize value of type `i8` using Schema::Long: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ -57i16,
+ &schema,
+ &names,
+ "Failed to serialize value of type `i16` using Schema::Long: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 129i32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `i32` using Schema::Long: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 24u64,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `u64` using Schema::Long: Expected Schema::Fixed(name: "u64", size: 8)"#,
+ );
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Long: Expected Schema::String | Schema::Uuid(String)",
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Long: Expected Schema::Union([Schema::Null, _])",
);
Ok(())
@@ -870,55 +937,60 @@ mod tests {
#[test]
fn test_serialize_float() -> TestResult {
let schema = Schema::Float;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 4.7f32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- (-14.1f64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize(4.7f32, &schema, &names, &[102, 102, 150, 64]);
+ assert_serialize_err(
+ -14.1f64,
+ &schema,
+ &names,
+ "Failed to serialize value of type `f64` using Schema::Float: Expected Schema::Double",
+ );
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Float: Expected Schema::String | Schema::Uuid(String)",
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Float: Expected Schema::Union([Schema::Null, _])",
);
- assert_eq!(buffer.as_slice(), &[102, 102, 150, 64, 154, 153, 97, 193]);
-
Ok(())
}
#[test]
fn test_serialize_double() -> TestResult {
- let schema = Schema::Float;
- let mut buffer: Vec<u8> = Vec::new();
+ let schema = Schema::Double;
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 4.7f32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- (-14.1f64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize(
+ -14.1f64,
+ &schema,
+ &names,
+ &[51, 51, 51, 51, 51, 51, 44, 192],
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ 4.7f32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `f32` using Schema::Double: Expected Schema::Float",
+ );
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Double: Expected Schema::String | Schema::Uuid(String)",
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Double: Expected Schema::Union([Schema::Null, _])",
);
-
- assert_eq!(buffer.as_slice(), &[102, 102, 150, 64, 154, 153, 97, 193]);
Ok(())
}
@@ -926,34 +998,43 @@ mod tests {
#[test]
fn test_serialize_bytes() -> TestResult {
let schema = Schema::Bytes;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- "test".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- Bytes::new(&[12, 3, 7, 91, 4]).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ assert_serialize(
+ Bytes::new(&[12, 3, 7, 91, 4]),
&schema,
- config,
- )?)?;
- assert!(
- ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ &names,
+ &[10, 12, 3, 7, 91, 4],
);
- assert!(
- PhantomData::<String>
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ 'a',
+ &schema,
+ &names,
+ "Failed to serialize value of type `char` using Schema::Bytes: Expected Schema::String",
);
-
- assert_eq!(
- buffer.as_slice(),
- &[2, b'a', 8, b't', b'e', b's', b't', 10, 12, 3, 7, 91, 4]
+ assert_serialize_err(
+ "test",
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Bytes: Expected Schema::String | Schema::Uuid(String)",
+ );
+ assert_serialize_err(
+ (),
+ &schema,
+ &names,
+ "Failed to serialize value of type `unit` using Schema::Bytes: Expected Schema::Null",
+ );
+ assert_serialize_err(
+ PhantomData::<String>,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `unit struct` using Schema::Bytes: Expected Schema::Record(name: "PhantomData", fields: [])"#,
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::Bytes: Expected Schema::Union([Schema::Null, _])",
);
Ok(())
@@ -962,34 +1043,39 @@ mod tests {
#[test]
fn test_serialize_string() -> TestResult {
let schema = Schema::String;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- "test".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- Bytes::new(&[12, 3, 7, 91, 4]).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ assert_serialize('a', &schema, &names, &[2, b'a']);
+ assert_serialize("test", &schema, &names, &[8, b't', b'e', b's', b't']);
+ assert_serialize(
+ BigDecimal::new(BigInt::new(Sign::Plus, vec![50024]), 2),
&schema,
- config,
- )?)?;
- assert!(
- ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ &names,
+ &[12, b'5', b'0', b'0', b'.', b'2', b'4'],
);
- assert!(
- PhantomData::<String>
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ Bytes::new(&[12, 3, 7, 91, 4]),
+ &schema,
+ &names,
+ "Failed to serialize value of type `bytes` using Schema::String: Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Bytes | Fixed) | Schema::Duration",
);
-
- assert_eq!(
- buffer.as_slice(),
- &[2, b'a', 8, b't', b'e', b's', b't', 10, 12, 3, 7, 91, 4]
+ assert_serialize_err(
+ (),
+ &schema,
+ &names,
+ "Failed to serialize value of type `unit` using Schema::String: Expected Schema::Null",
+ );
+ assert_serialize_err(
+ PhantomData::<String>,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `unit struct` using Schema::String: Expected Schema::Record(name: "PhantomData", fields: [])"#,
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ "Failed to serialize value of type `some` using Schema::String: Expected Schema::Union([Schema::Null, _])",
);
Ok(())
@@ -1009,55 +1095,55 @@ mod tests {
)?;
#[derive(Serialize)]
- #[serde(rename_all = "camelCase")]
+ #[serde(rename_all = "camelCase", rename = "TestRecord")]
struct GoodTestRecord {
string_field: String,
int_field: i32,
}
#[derive(Serialize)]
- #[serde(rename_all = "camelCase")]
+ #[serde(rename_all = "camelCase", rename = "TestRecord")]
struct BadTestRecord {
foo_string_field: String,
bar_int_field: i32,
}
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
let good_record = GoodTestRecord {
string_field: String::from("test"),
int_field: 10,
};
- good_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(
+ good_record,
+ &schema,
+ &names,
+ &[8, b't', b'e', b's', b't', 20],
+ );
let bad_record = BadTestRecord {
foo_string_field: String::from("test"),
bar_int_field: 10,
};
- assert!(
- bad_record
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ bad_record,
+ &schema,
+ &names,
+ r#"Missing field in record: "fooStringField""#,
);
-
- assert!(
- "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `str` using Schema::Record(RecordSchema { name: Name { name: "TestRecord", .. }, fields: [RecordField { name: "stringField", schema: String, .. }, RecordField { name: "intField", schema: Int, .. }], .. }): Expected Schema::String | Schema::Uuid(String)"#,
);
- assert!(
- Some("")
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .is_err()
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `some` using Schema::Record(RecordSchema { name: Name { name: "TestRecord", .. }, fields: [RecordField { name: "stringField", schema: String, .. }, RecordField { name: "intField", schema: Int, .. }], .. }): Expected Schema::Union([Schema::Null, _])"#,
);
- assert_eq!(buffer.as_slice(), &[8, b't', b'e', b's', b't', 20]);
-
Ok(())
}
@@ -1071,47 +1157,45 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
#[derive(Serialize)]
struct EmptyRecord;
- EmptyRecord.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(EmptyRecord, &schema, &names, &[]);
#[derive(Serialize)]
+ #[serde(rename = "EmptyRecord")]
struct NonEmptyRecord {
foo: String,
}
let record = NonEmptyRecord {
foo: "bar".to_string(),
};
- match record
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::FieldName(field_name)) if field_name == "foo" => (),
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "none"); // serialize_unit() delegates to serialize_none()
- assert_eq!(value, "None. Cause: Expected: Record. Got: Null");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.len(), 0);
+ assert_serialize_err(record, &schema, &names, r#"Missing field in record: "foo""#);
+ assert_serialize_err(
+ (),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `unit` using Schema::Record(RecordSchema { name: Name { name: "EmptyRecord", .. }, fields: [], .. }): Expected Schema::Null"#,
+ );
+ assert_serialize_err(
+ "",
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `str` using Schema::Record(RecordSchema { name: Name { name: "EmptyRecord", .. }, fields: [], .. }): Expected Schema::String | Schema::Uuid(String)"#,
+ );
+ assert_serialize_err(
+ PhantomData::<String>,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `unit struct` using Schema::Record(RecordSchema { name: Name { name: "EmptyRecord", .. }, fields: [], .. }): Expected Schema::Record(name: "PhantomData", fields: [])"#,
+ );
+ assert_serialize_err(
+ Some(""),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `some` using Schema::Record(RecordSchema { name: Name { name: "EmptyRecord", .. }, fields: [], .. }): Expected Schema::Union([Schema::Null, _])"#,
+ );
Ok(())
}
@@ -1127,6 +1211,7 @@ mod tests {
)?;
#[derive(Serialize)]
+ #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
enum Suit {
Spades,
Hearts,
@@ -1134,35 +1219,18 @@ mod tests {
Clubs,
}
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- Suit::Spades.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- Suit::Hearts.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- Suit::Diamonds.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- Suit::Clubs.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- match None::<()>
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "none");
- assert_eq!(value, "None. Cause: Expected: Enum. Got: Null");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
- assert_eq!(buffer.as_slice(), &[0, 2, 4, 6]);
+ assert_serialize(Suit::Spades, &schema, &names, &[0]);
+ assert_serialize(Suit::Hearts, &schema, &names, &[2]);
+ assert_serialize(Suit::Diamonds, &schema, &names, &[4]);
+ assert_serialize(Suit::Clubs, &schema, &names, &[6]);
+ assert_serialize_err(
+ None::<()>,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `none` using Schema::Enum(EnumSchema { name: Name { name: "Suit", .. }, symbols: ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"], .. }): Expected Schema::Union([Schema::Null, _])"#,
+ );
Ok(())
}
@@ -1176,34 +1244,20 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- let arr: Vec<i64> = vec![10, 5, 400];
- arr.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match vec![1_f32]
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "f32");
- assert_eq!(value, "1. Cause: Expected: Long. Got: Float");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[6, 20, 10, 160, 6, 0]);
+ assert_serialize(
+ vec![10i64, 5, 400],
+ &schema,
+ &names,
+ &[6, 20, 10, 160, 6, 0],
+ );
+ assert_serialize_err(
+ vec![1_f32],
+ &schema,
+ &names,
+ "Failed to serialize value of type `f32` using Schema::Long: Expected Schema::Float",
+ );
Ok(())
}
@@ -1217,44 +1271,29 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
let mut map: BTreeMap<String, i64> = BTreeMap::new();
map.insert(String::from("item1"), 10);
map.insert(String::from("item2"), 400);
- map.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(
+ map,
+ &schema,
+ &names,
+ &[
+ 4, 10, b'i', b't', b'e', b'm', b'1', 20, 10, b'i', b't', b'e', b'm', b'2', 160, 6,
+ 0,
+ ],
+ );
let mut map: BTreeMap<String, &str> = BTreeMap::new();
map.insert(String::from("item1"), "value1");
- match map
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "string");
- assert_eq!(value, "value1. Cause: Expected: Long. Got: String");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(
- buffer.as_slice(),
- &[
- 4, 10, b'i', b't', b'e', b'm', b'1', 20, 10, b'i', b't', b'e', b'm', b'2', 160, 6,
- 0
- ]
+ assert_serialize_err(
+ map,
+ &schema,
+ &names,
+ "Failed to serialize value of type `str` using Schema::Long: Expected Schema::String | Schema::Uuid(String)",
);
Ok(())
@@ -1274,43 +1313,20 @@ mod tests {
Long(i64),
}
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- Some(10i64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- None::<i64>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- NullableLong::Long(400).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ assert_serialize(Some(10i64), &schema, &names, &[2, 20]);
+ assert_serialize(None::<i64>, &schema, &names, &[0]);
+ assert_serialize(NullableLong::Long(400), &schema, &names, &[2, 160, 6]);
+ assert_serialize(NullableLong::Null, &schema, &names, &[0]);
+ assert_serialize(400i64, &schema, &names, &[2, 160, 6]);
+ assert_serialize((), &schema, &names, &[0]);
+ assert_serialize_err(
+ "invalid",
&schema,
- config,
- )?)?;
- NullableLong::Null.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match "invalid"
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "string");
- assert_eq!(
- value,
- "invalid. Cause: Expected one of the union variants [Null, Long]. Got: String"
- );
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[2, 20, 0, 2, 160, 6, 0]);
+ &names,
+ "Failed to serialize value of type `str` using Schema::Union(UnionSchema { schemas: [Null, Long] }): Expected Schema::String in variants",
+ );
Ok(())
}
@@ -1330,48 +1346,23 @@ mod tests {
Str(String),
}
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- LongOrString::Null.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- LongOrString::Long(400).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ assert_serialize(LongOrString::Null, &schema, &names, &[0]);
+ assert_serialize(LongOrString::Long(400), &schema, &names, &[2, 160, 6]);
+ assert_serialize(
+ LongOrString::Str("test".into()),
&schema,
- config,
- )?)?;
- LongOrString::Str(String::from("test")).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ &names,
+ &[4, 8, b't', b'e', b's', b't'],
+ );
+ assert_serialize((), &schema, &names, &[0]);
+ assert_serialize(400i64, &schema, &names, &[2, 160, 6]);
+ assert_serialize("test", &schema, &names, &[4, 8, b't', b'e', b's', b't']);
+ assert_serialize_err(
+ 1f64,
&schema,
- config,
- )?)?;
-
- match 1_f64
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "f64");
- assert_eq!(
- value,
- "1. Cause: Cannot find a Double schema in [Null, Long, String]"
- );
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(
- buffer.as_slice(),
- &[0, 2, 160, 6, 4, 8, b't', b'e', b's', b't']
+ &names,
+ "Failed to serialize value of type `f64` using Schema::Union(UnionSchema { schemas: [Null, Long, String] }): Expected Schema::Double in variants",
);
Ok(())
@@ -1387,75 +1378,32 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- Bytes::new(&[10, 124, 31, 97, 14, 201, 3, 88]).serialize(SchemaAwareSerializer::new(
- &mut buffer,
+ assert_serialize(
+ Bytes::new(&[10, 124, 31, 97, 14, 201, 3, 88]),
&schema,
- config,
- )?)?;
-
- // non-8 size
- match Bytes::new(&[123])
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "bytes");
- assert_eq!(
- value,
- "7b. Cause: Fixed schema size (8) does not match the value length (1)"
- ); // Bytes represents its values as hexadecimals: '7b' is 123
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- // array
- match [1; 8]
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "tuple"); // TODO: why is this 'tuple' ?!
- assert_eq!(value, "tuple (len=8). Cause: Expected: Fixed. Got: Array");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- // slice
- match &[1, 2, 3, 4, 5, 6, 7, 8]
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(*value_type, "tuple"); // TODO: why is this 'tuple' ?!
- assert_eq!(value, "tuple (len=8). Cause: Expected: Fixed. Got: Array");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[10, 124, 31, 97, 14, 201, 3, 88]);
+ &names,
+ &[10, 124, 31, 97, 14, 201, 3, 88],
+ );
+ assert_serialize_err(
+ Bytes::new(&[123]),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `bytes` using Schema::Fixed(FixedSchema { name: Name { name: "LongVal", .. }, size: 8, .. }): Fixed size (8) does not match bytes length (1)"#,
+ );
+ assert_serialize_err(
+ [1u8; 8],
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `tuple` using Schema::Fixed(FixedSchema { name: Name { name: "LongVal", .. }, size: 8, .. }): Expected Schema::Record(fields.len() == 8)"#,
+ );
+ assert_serialize_err(
+ [1u8, 2, 3, 4, 5, 6, 7, 8].as_slice(),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `seq` using Schema::Fixed(FixedSchema { name: Name { name: "LongVal", .. }, size: 8, .. }): Expected Schema::Array"#,
+ );
Ok(())
}
@@ -1471,31 +1419,16 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
let val = Decimal::from(&[251, 155]);
- val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "none");
- assert_eq!(value, "None. Cause: Expected: Decimal. Got: Null");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[4, 251, 155]);
+ assert_serialize(val, &schema, &names, &[4, 251, 155]);
+ assert_serialize_err(
+ (),
+ &schema,
+ &names,
+ "Failed to serialize value of type `unit` using Schema::Decimal(DecimalSchema { precision: 16, scale: 2, inner: Bytes }): Expected Schema::Null",
+ );
Ok(())
}
@@ -1513,31 +1446,16 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
let val = Decimal::from(&[0, 0, 0, 0, 0, 0, 251, 155]);
- val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "none");
- assert_eq!(value, "None. Cause: Expected: Decimal. Got: Null");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[0, 0, 0, 0, 0, 0, 251, 155]);
+ assert_serialize(val, &schema, &names, &[0, 0, 0, 0, 0, 0, 251, 155]);
+ assert_serialize_err(
+ (),
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `unit` using Schema::Decimal(DecimalSchema { precision: 16, scale: 2, inner: Fixed(FixedSchema { name: Name { name: "FixedDecimal", .. }, size: 8, attributes: {"precision": Number(16), "scale": Number(2)}, .. }) }): Expected Schema::Null"#,
+ );
Ok(())
}
@@ -1551,18 +1469,21 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- let val = BigDecimal::new(BigInt::new(Sign::Plus, vec![50024]), 2);
- val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ #[derive(Serialize)]
+ #[serde(transparent)]
+ struct BigDecimalWrapper {
+ // This is needed because the Serialize implementation of BigDecimal serializes to a string.
+ // The with implementation serializes to bytes.
+ #[serde(with = "crate::serde::bigdecimal")]
+ value: BigDecimal,
+ }
- assert_eq!(buffer.as_slice(), &[10, 6, 0, 195, 104, 4]);
+ let val = BigDecimalWrapper {
+ value: BigDecimal::new(BigInt::new(Sign::Plus, vec![50024]), 2),
+ };
+ assert_serialize(val, &schema, &names, &[10, 6, 0, 195, 104, 4]);
Ok(())
}
@@ -1578,40 +1499,25 @@ mod tests {
}"#,
)?;
+ // Uuid serialize implementation changes based on this value
assert!(!crate::util::is_human_readable());
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- "8c28da81-238c-4326-bddd-4e3d00cc5099"
- .parse::<Uuid>()?
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ let uuid = "8c28da81-238c-4326-bddd-4e3d00cc5099".parse::<Uuid>()?;
- match 1_u8
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "u8");
- assert_eq!(value, "1. Cause: Expected: Uuid. Got: Int");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(
- buffer.as_slice(),
+ assert_serialize(
+ uuid,
+ &schema,
+ &names,
&[
- 140, 40, 218, 129, 35, 140, 67, 38, 189, 221, 78, 61, 0, 204, 80, 153
- ]
+ 140, 40, 218, 129, 35, 140, 67, 38, 189, 221, 78, 61, 0, 204, 80, 153,
+ ],
+ );
+ assert_serialize_err(
+ 1u8,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `u8` using Schema::Uuid(Fixed(FixedSchema { name: Name { name: "FixedUuid", .. }, size: 16, .. })): Expected Schema::Int | Schema::Date | Schema::TimeMillis"#,
);
Ok(())
@@ -1626,39 +1532,23 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match 10000_f32
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "f32");
- assert_eq!(value, "10000. Cause: Expected: Date. Got: Float");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
- assert_eq!(
- buffer.as_slice(),
- &[200, 1, 208, 15, 160, 156, 1, 208, 15, 160, 156, 1]
+ assert_serialize(100u8, &schema, &names, &[200, 1]);
+ assert_serialize(1000u16, &schema, &names, &[208, 15]);
+ assert_serialize(1000i16, &schema, &names, &[208, 15]);
+ assert_serialize(10000i32, &schema, &names, &[160, 156, 1]);
+ assert_serialize_err(
+ 10000u32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u32` using Schema::Date: Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ );
+ assert_serialize_err(
+ 10000f32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `f32` using Schema::Date: Expected Schema::Float",
);
Ok(())
@@ -1673,39 +1563,23 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- match 10000_f32
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "f32");
- assert_eq!(value, "10000. Cause: Expected: TimeMillis. Got: Float");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(
- buffer.as_slice(),
- &[200, 1, 208, 15, 160, 156, 1, 208, 15, 160, 156, 1]
+ assert_serialize(100u8, &schema, &names, &[200, 1]);
+ assert_serialize(1000u16, &schema, &names, &[208, 15]);
+ assert_serialize(1000i16, &schema, &names, &[208, 15]);
+ assert_serialize(10000i32, &schema, &names, &[160, 156, 1]);
+ assert_serialize_err(
+ 10000u32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u32` using Schema::TimeMillis: Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ );
+ assert_serialize_err(
+ 10000f32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `f32` using Schema::TimeMillis: Expected Schema::Float",
);
Ok(())
@@ -1720,42 +1594,39 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match 10000_f32
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "f32");
- assert_eq!(value, "10000. Cause: Expected: TimeMicros. Got: Float");
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
- assert_eq!(
- buffer.as_slice(),
- &[
- 200, 1, 208, 15, 160, 156, 1, 208, 15, 160, 156, 1, 160, 156, 1
- ]
+ assert_serialize(10000u32, &schema, &names, &[160, 156, 1]);
+ assert_serialize(10000i64, &schema, &names, &[160, 156, 1]);
+ assert_serialize_err(
+ 100u8,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u8` using Schema::TimeMicros: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 1000u16,
+ &schema,
+ &names,
+ "Failed to serialize value of type `u16` using Schema::TimeMicros: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 1000i16,
+ &schema,
+ &names,
+ "Failed to serialize value of type `i16` using Schema::TimeMicros: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 10000i32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `i32` using Schema::TimeMicros: Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ );
+ assert_serialize_err(
+ 10000f32,
+ &schema,
+ &names,
+ "Failed to serialize value of type `f32` using Schema::TimeMicros: Expected Schema::Float",
);
Ok(())
@@ -1763,7 +1634,11 @@ mod tests {
#[test]
fn test_serialize_timestamp() -> TestResult {
- for precision in ["millis", "micros", "nanos"] {
+ for (precision, error) in [
+ ("millis", "Millis"),
+ ("micros", "Micros"),
+ ("nanos", "Nanos"),
+ ] {
let schema = Schema::parse_str(&format!(
r#"{{
"type": "long",
@@ -1771,50 +1646,49 @@ mod tests {
}}"#
))?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- 10000_i64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match 10000_f64
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- let mut capital_precision = precision.to_string();
- if let Some(c) = capital_precision.chars().next() {
- capital_precision.replace_range(..1, &c.to_uppercase().to_string());
- }
- assert_eq!(value_type, "f64");
- assert_eq!(
- value,
- format!(
- "10000. Cause: Expected: Timestamp{capital_precision}. Got: Double"
- )
- );
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
- assert_eq!(
- buffer.as_slice(),
- &[
- 200, 1, 208, 15, 160, 156, 1, 208, 15, 160, 156, 1, 160, 156, 1
- ]
+ assert_serialize(10000u32, &schema, &names, &[160, 156, 1]);
+ assert_serialize(10000i64, &schema, &names, &[160, 156, 1]);
+ assert_serialize_err(
+ 100u8,
+ &schema,
+ &names,
+ &format!(
+ "Failed to serialize value of type `u8` using Schema::Timestamp{error}: Expected Schema::Int | Schema::Date | Schema::TimeMillis"
+ ),
+ );
+ assert_serialize_err(
+ 1000u16,
+ &schema,
+ &names,
+ &format!(
+ "Failed to serialize value of type `u16` using Schema::Timestamp{error}: Expected Schema::Int | Schema::Date | Schema::TimeMillis"
+ ),
+ );
+ assert_serialize_err(
+ 1000i16,
+ &schema,
+ &names,
+ &format!(
+ "Failed to serialize value of type `i16` using Schema::Timestamp{error}: Expected Schema::Int | Schema::Date | Schema::TimeMillis"
+ ),
+ );
+ assert_serialize_err(
+ 10000i32,
+ &schema,
+ &names,
+ &format!(
+ "Failed to serialize value of type `i32` using Schema::Timestamp{error}: Expected Schema::Int | Schema::Date | Schema::TimeMillis"
+ ),
+ );
+ assert_serialize_err(
+ 10000f32,
+ &schema,
+ &names,
+ &format!(
+ "Failed to serialize value of type `f32` using Schema::Timestamp{error}: Expected Schema::Float"
+ ),
);
}
@@ -1832,38 +1706,21 @@ mod tests {
}"#,
)?;
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- let duration_bytes =
- ByteArray::new(Duration::new(Months::new(3), Days::new(2), Millis::new(1200)).into());
- duration_bytes.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- match [1; 12]
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .map_err(Error::into_details)
- {
- Err(Details::SerializeValueWithSchema {
- value_type,
- value,
- schema,
- }) => {
- assert_eq!(value_type, "tuple"); // TODO: why is this 'tuple' ?!
- assert_eq!(
- value,
- "tuple (len=12). Cause: Expected: Duration. Got: Array"
- );
- assert_eq!(schema, schema);
- }
- unexpected => panic!("Expected an error. Got: {unexpected:?}"),
- }
-
- assert_eq!(buffer.as_slice(), &[3, 0, 0, 0, 2, 0, 0, 0, 176, 4, 0, 0]);
+ let duration = Duration::new(Months::new(3), Days::new(2), Millis::new(1200));
+ assert_serialize(
+ duration,
+ &schema,
+ &names,
+ &[3, 0, 0, 0, 2, 0, 0, 0, 176, 4, 0, 0],
+ );
+ assert_serialize_err(
+ [0u8; 12],
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `tuple` using Schema::Duration(FixedSchema { name: Name { name: "duration", .. }, size: 12, .. }): Expected Schema::Record(fields.len() == 12)"#,
+ );
Ok(())
}
@@ -1877,7 +1734,6 @@ mod tests {
"fields": [
{"name": "stringField", "type": "string"},
{"name": "intField", "type": "int"},
- {"name": "bigDecimalField", "type": {"type": "bytes", "logicalType": "big-decimal"}},
{"name": "uuidField", "type": {"name": "uuid", "type": "fixed", "size": 16, "logicalType": "uuid"}},
{"name": "innerRecord", "type": ["null", "TestRecord"]}
]
@@ -1889,44 +1745,34 @@ mod tests {
struct TestRecord {
string_field: String,
int_field: i32,
- big_decimal_field: BigDecimal,
uuid_field: Uuid,
// #[serde(skip_serializing_if = "Option::is_none")] => Never ignore None!
inner_record: Option<Box<TestRecord>>,
}
assert!(!crate::util::is_human_readable());
- let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let config = Config {
- names: rs.get_names(),
- target_block_size: None,
- human_readable: false,
- };
let good_record = TestRecord {
string_field: String::from("test"),
int_field: 10,
- big_decimal_field: BigDecimal::new(BigInt::new(Sign::Plus, vec![50024]), 2),
uuid_field: "8c28da81-238c-4326-bddd-4e3d00cc5098".parse::<Uuid>()?,
inner_record: Some(Box::new(TestRecord {
string_field: String::from("inner_test"),
int_field: 100,
- big_decimal_field: BigDecimal::new(BigInt::new(Sign::Plus, vec![20038]), 2),
uuid_field: "8c28da81-238c-4326-bddd-4e3d00cc5099".parse::<Uuid>()?,
inner_record: None,
})),
};
- good_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- assert_eq!(
- buffer.as_slice(),
+ assert_serialize(
+ good_record,
+ &schema,
+ rs.get_names(),
&[
- 8, 116, 101, 115, 116, 20, 10, 6, 0, 195, 104, 4, 140, 40, 218, 129, 35, 140, 67,
- 38, 189, 221, 78, 61, 0, 204, 80, 152, 2, 20, 105, 110, 110, 101, 114, 95, 116,
- 101, 115, 116, 200, 1, 8, 4, 78, 70, 4, 140, 40, 218, 129, 35, 140, 67, 38, 189,
- 221, 78, 61, 0, 204, 80, 153, 0
- ]
+ 8, 116, 101, 115, 116, 20, 140, 40, 218, 129, 35, 140, 67, 38, 189, 221, 78, 61, 0,
+ 204, 80, 152, 2, 20, 105, 110, 110, 101, 114, 95, 116, 101, 115, 116, 200, 1, 140,
+ 40, 218, 129, 35, 140, 67, 38, 189, 221, 78, 61, 0, 204, 80, 153, 0,
+ ],
);
Ok(())
@@ -1980,34 +1826,43 @@ mod tests {
bar: String,
}
- let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let config = Config {
- names: rs.get_names(),
- target_block_size: None,
- human_readable: false,
- };
let foo_record = TestRecord {
inner_union: InnerUnion::InnerVariantFoo(InnerRecordFoo {
foo: String::from("foo"),
}),
};
- foo_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(
+ foo_record,
+ &schema,
+ rs.get_names(),
+ &[0, 6, b'f', b'o', b'o'],
+ );
let bar_record = TestRecord {
inner_union: InnerUnion::InnerVariantBar(InnerRecordBar {
bar: String::from("bar"),
}),
};
- bar_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(
+ bar_record,
+ &schema,
+ rs.get_names(),
+ &[2, 6, b'b', b'a', b'r'],
+ );
let int_record = TestRecord {
inner_union: InnerUnion::IntField(1),
};
- int_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(int_record, &schema, rs.get_names(), &[4, 2]);
let string_record = TestRecord {
inner_union: InnerUnion::StringField(String::from("string")),
};
- string_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize(
+ string_record,
+ &schema,
+ rs.get_names(),
+ &[6, 12, b's', b't', b'r', b'i', b'n', b'g'],
+ );
Ok(())
}
@@ -2042,54 +1897,28 @@ mod tests {
#[derive(Serialize)]
#[serde(untagged)]
enum InnerUnion {
- InnerVariantFoo(InnerRecordFoo),
- InnerVariantBar(InnerRecordBar),
IntField(i32),
- StringField(String),
- }
-
- #[derive(Serialize)]
- #[serde(rename = "innerRecordFoo")]
- struct InnerRecordFoo {
- foo: String,
}
- #[derive(Serialize)]
- #[serde(rename = "innerRecordBar")]
- struct InnerRecordBar {
- bar: String,
- }
-
- let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let config = Config {
- names: rs.get_names(),
- target_block_size: None,
- human_readable: false,
- };
+ // Flattening a Option into the underlying union is NOT supported
let null_record = TestRecord { inner_union: None };
- null_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize_err(
+ null_record,
+ &schema,
+ rs.get_names(),
+ r#"Failed to serialize field 'innerUnion' of record RecordSchema { name: Name { name: "TestRecord", .. }, fields: [RecordField { name: "innerUnion", schema: Union(UnionSchema { schemas: [Null, Record(RecordSchema { name: Name { name: "innerRecordFoo", .. }, fields: [RecordField { name: "foo", schema: String, .. }], .. }), Record(RecordSchema { name: Name { name: "innerRecordBar", .. }, fields: [RecordField { name: "bar", schema: String, .. }], .. }), Int, String] }), .. }], . [...]
+ );
let foo_record = TestRecord {
- inner_union: Some(InnerUnion::InnerVariantFoo(InnerRecordFoo {
- foo: String::from("foo"),
- })),
+ inner_union: Some(InnerUnion::IntField(42)),
};
- foo_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- let bar_record = TestRecord {
- inner_union: Some(InnerUnion::InnerVariantBar(InnerRecordBar {
- bar: String::from("bar"),
- })),
- };
- bar_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- let int_record = TestRecord {
- inner_union: Some(InnerUnion::IntField(1)),
- };
- int_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- let string_record = TestRecord {
- inner_union: Some(InnerUnion::StringField(String::from("string"))),
- };
- string_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert_serialize_err(
+ foo_record,
+ &schema,
+ rs.get_names(),
+ r#"Failed to serialize field 'innerUnion' of record RecordSchema { name: Name { name: "TestRecord", .. }, fields: [RecordField { name: "innerUnion", schema: Union(UnionSchema { schemas: [Null, Record(RecordSchema { name: Name { name: "innerRecordFoo", .. }, fields: [RecordField { name: "foo", schema: String, .. }], .. }), Record(RecordSchema { name: Name { name: "innerRecordBar", .. }, fields: [RecordField { name: "bar", schema: String, .. }], .. }), Int, String] }), .. }], . [...]
+ );
Ok(())
}
@@ -2099,9 +1928,6 @@ mod tests {
struct Foo {
a: String,
b: String,
- c: usize,
- d: f64,
- e: usize,
}
let schema = Schema::parse_str(
r#"
@@ -2116,65 +1942,34 @@ mod tests {
{
"name":"a",
"type":"string"
- },
- {
- "name":"d",
- "type":"double"
- },
- {
- "name":"e",
- "type":"long"
- },
- {
- "name":"c",
- "type":"long"
}
]
}
"#,
)?;
- let mut writer = Writer::new(&schema, Vec::new())?;
- writer.append_ser(Foo {
+ let names = HashMap::new();
+ let foo = Foo {
a: "Hello".into(),
b: "World".into(),
- c: 42,
- d: std::f64::consts::PI,
- e: 5,
- })?;
- let encoded = writer.into_inner()?;
- let mut reader = Reader::builder(&encoded[..])
- .reader_schema(&schema)
- .build()?;
- let decoded = from_value::<Foo>(&reader.next().unwrap()?)?;
- assert_eq!(
- decoded,
- Foo {
- a: "Hello".into(),
- b: "World".into(),
- c: 42,
- d: std::f64::consts::PI,
- e: 5
- }
+ };
+ // Serializing fields out of order is NOT supported
+ assert_serialize_err(
+ foo,
+ &schema,
+ &names,
+ r#"Missing default for skipped field 'b' of schema RecordSchema { name: Name { name: "Foo", .. }, fields: [RecordField { name: "b", schema: String, .. }, RecordField { name: "a", schema: String, .. }], .. }"#,
);
+
Ok(())
}
#[test]
fn avro_rs_414_serialize_char_as_string() -> TestResult {
let schema = Schema::String;
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert_eq!(buffer.as_slice(), &[2, b'a']);
+ assert_serialize('a', &schema, &names, &[2, b'a']);
Ok(())
}
@@ -2182,18 +1977,14 @@ mod tests {
#[test]
fn avro_rs_414_serialize_char_as_bytes() -> TestResult {
let schema = Schema::Bytes;
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- assert_eq!(buffer.as_slice(), &[2, b'a']);
+ assert_serialize_err(
+ 'a',
+ &schema,
+ &names,
+ "Failed to serialize value of type `char` using Schema::Bytes: Expected Schema::String",
+ );
Ok(())
}
@@ -2207,18 +1998,14 @@ mod tests {
size: 4,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- assert_eq!(buffer.as_slice(), &[b'a', 0, 0, 0]);
+ assert_serialize_err(
+ 'a',
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `char` using Schema::Fixed(FixedSchema { name: Name { name: "char", .. }, size: 4, .. }): Expected Schema::String"#,
+ );
Ok(())
}
@@ -2226,120 +2013,9 @@ mod tests {
#[test]
fn avro_rs_414_serialize_emoji_char_as_string() -> TestResult {
let schema = Schema::String;
-
- let mut buffer: Vec<u8> = Vec::new();
- let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- assert_eq!(buffer.as_slice(), &[8, 240, 159, 145, 185]);
-
- Ok(())
- }
-
- #[test]
- fn avro_rs_414_serialize_emoji_char_as_bytes() -> TestResult {
- let schema = Schema::Bytes;
-
- let mut buffer: Vec<u8> = Vec::new();
- let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- assert_eq!(buffer.as_slice(), &[8, 240, 159, 145, 185]);
-
- Ok(())
- }
-
- #[test]
- fn avro_rs_414_serialize_emoji_char_as_fixed() -> TestResult {
- let schema = Schema::Fixed(FixedSchema {
- name: Name::new("char")?,
- aliases: None,
- doc: None,
- size: 4,
- attributes: Default::default(),
- });
-
- let mut buffer: Vec<u8> = Vec::new();
- let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
-
- // This is a different byte value than the tests above. This is because by creating a String
- // the unicode value is normalized by Rust
- assert_eq!(buffer.as_slice(), &[121, 244, 1, 0]);
-
- Ok(())
- }
-
- #[test]
- fn avro_rs_414_serialize_char_as_fixed_wrong_name() -> TestResult {
- let schema = Schema::Fixed(FixedSchema {
- name: Name::new("characters")?,
- aliases: None,
- doc: None,
- size: 4,
- attributes: Default::default(),
- });
-
- let mut buffer: Vec<u8> = Vec::new();
- let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- assert!(matches!(
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
-
- Ok(())
- }
-
- #[test]
- fn avro_rs_414_serialize_char_as_fixed_wrong_size() -> TestResult {
- let schema = Schema::Fixed(FixedSchema {
- name: Name::new("char")?,
- aliases: None,
- doc: None,
- size: 1,
- attributes: Default::default(),
- });
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- assert!(matches!(
- 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
+ assert_serialize('👹', &schema, &names, &[8, 240, 159, 145, 185]);
Ok(())
}
@@ -2353,25 +2029,16 @@ mod tests {
size: 16,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
-
- let bytes_written =
- i128::MAX.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert_eq!(bytes_written, 16);
- assert_eq!(
- buffer.as_slice(),
+ assert_serialize(
+ i128::MAX,
+ &schema,
+ &names,
&[
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
- 0xFF, 0x7F
- ]
+ 0xFF, 0x7F,
+ ],
);
Ok(())
@@ -2386,22 +2053,14 @@ mod tests {
size: 16,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- assert!(matches!(
- i128::MAX
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
+ assert_serialize_err(
+ i128::MAX,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `i128` using Schema::Fixed(FixedSchema { name: Name { name: "onehundredtwentyeight", .. }, size: 16, .. }): Expected Schema::Fixed(name: "i128", size: 16)"#,
+ );
Ok(())
}
@@ -2415,22 +2074,14 @@ mod tests {
size: 8,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- assert!(matches!(
- i128::MAX
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
+ assert_serialize_err(
+ i128::MAX,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `i128` using Schema::Fixed(FixedSchema { name: Name { name: "i128", .. }, size: 8, .. }): Expected Schema::Fixed(name: "i128", size: 16)"#,
+ );
Ok(())
}
@@ -2444,25 +2095,16 @@ mod tests {
size: 16,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- let bytes_written =
- u128::MAX.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- assert_eq!(bytes_written, 16);
-
- assert_eq!(
- buffer.as_slice(),
+ assert_serialize(
+ u128::MAX,
+ &schema,
+ &names,
&[
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
- 0xFF, 0xFF
- ]
+ 0xFF, 0xFF,
+ ],
);
Ok(())
@@ -2477,22 +2119,14 @@ mod tests {
size: 16,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- assert!(matches!(
- u128::MAX
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
+ assert_serialize_err(
+ u128::MAX,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `u128` using Schema::Fixed(FixedSchema { name: Name { name: "onehundredtwentyeight", .. }, size: 16, .. }): Expected Schema::Fixed(name: "u128", size: 16)"#,
+ );
Ok(())
}
@@ -2506,22 +2140,14 @@ mod tests {
size: 8,
attributes: Default::default(),
});
-
- let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- assert!(matches!(
- u128::MAX
- .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
- .unwrap_err()
- .details(),
- Details::SerializeValueWithSchema { .. }
- ));
+ assert_serialize_err(
+ u128::MAX,
+ &schema,
+ &names,
+ r#"Failed to serialize value of type `u128` using Schema::Fixed(FixedSchema { name: Name { name: "u128", .. }, size: 8, .. }): Expected Schema::Fixed(name: "u128", size: 16)"#,
+ );
Ok(())
}
@@ -2533,28 +2159,15 @@ mod tests {
{ "name": "fixed4", "type": "fixed", "size": 4 },
{ "name": "fixed8", "type": "fixed", "size": 8 }
]"#,
- )
- .unwrap();
-
- let mut buffer: Vec<u8> = Vec::new();
- let names = HashMap::new();
- let config = Config::<'_, Schema> {
- names: &names,
- target_block_size: None,
- human_readable: false,
- };
- let bytes_written = crate::serde::fixed::serialize(
- &[0, 1, 2, 3],
- SchemaAwareSerializer::new(&mut buffer, &schema, config)?,
)?;
- assert_eq!(bytes_written, 4);
- let bytes_written = crate::serde::fixed::serialize(
- &[4, 5, 6, 7, 8, 9, 10, 11],
- SchemaAwareSerializer::new(&mut buffer, &schema, config)?,
- )?;
- assert_eq!(bytes_written, 8);
-
- assert_eq!(buffer, &[0, 0, 1, 2, 3, 2, 4, 5, 6, 7, 8, 9, 10, 11][..]);
+ let names = HashMap::new();
+ assert_serialize(Bytes::new(&[0, 1, 2, 3]), &schema, &names, &[0, 0, 1, 2, 3]);
+ assert_serialize(
+ Bytes::new(&[4, 5, 6, 7, 8, 9, 10, 11]),
+ &schema,
+ &names,
+ &[2, 4, 5, 6, 7, 8, 9, 10, 11],
+ );
Ok(())
}
diff --git a/avro/src/serde/ser_schema/record/mod.rs b/avro/src/serde/ser_schema/record/mod.rs
index 853867d..0d44828 100644
--- a/avro/src/serde/ser_schema/record/mod.rs
+++ b/avro/src/serde/ser_schema/record/mod.rs
@@ -53,6 +53,37 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
}
}
+ fn field_error(&self, position: usize, error: Error) -> Error {
+ let field = &self.record.fields[position];
+ let error = match error.into_details() {
+ Details::SerializeValueWithSchema {
+ value_type,
+ value,
+ schema: _,
+ } => format!("Failed to serialize value of type `{value_type}`: {value}"),
+ Details::SerializeRecordFieldWithSchema {
+ field_name,
+ record_schema,
+ error,
+ } => format!(
+ "Failed to serialize field '{field_name}' of record {}: {error}",
+ record_schema.name
+ ),
+ Details::MissingDefaultForSkippedField { field_name, schema } => {
+ format!(
+ "Missing default for skipped field '{field_name}' for record {}",
+ schema.name
+ )
+ }
+ details => format!("{details:?}"),
+ };
+ Error::new(Details::SerializeRecordFieldWithSchema {
+ field_name: field.name.clone(),
+ record_schema: self.record.clone(),
+ error,
+ })
+ }
+
fn serialize_next_field<T: ?Sized + Serialize>(
&mut self,
position: usize,
@@ -60,14 +91,16 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
) -> Result<(), Error> {
// Serialize any skipped fields using their default value
while self.field_position < position {
- self.serialize_default(position)?;
+ self.serialize_default(self.field_position)?;
}
let field = &self.record.fields[position];
- self.bytes_written += value.serialize(SchemaAwareSerializer::new(
- self.writer,
- &field.schema,
- self.config,
- )?)?;
+ self.bytes_written += value
+ .serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &field.schema,
+ self.config,
+ )?)
+ .map_err(|e| self.field_error(self.field_position, e))?;
self.field_position += 1;
Ok(())
@@ -77,21 +110,14 @@ impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
let field = &self.record.fields[position];
if let Some(default) = &field.default {
self.serialize_next_field(
- self.field_position,
+ position,
&SchemaAwareRecordFieldDefault::new(default, &field.schema),
)
- .map_err(|e| {
- Details::SerializeRecordFieldWithSchema {
- field_name: field.name.clone(),
- record_schema: Schema::Record(self.record.clone()),
- error: Box::new(e),
- }
- .into()
- })
+ .map_err(|e| self.field_error(position, e))
} else {
Err(Details::MissingDefaultForSkippedField {
field_name: field.name.clone(),
- schema: Schema::Record(self.record.clone()),
+ schema: self.record.clone(),
}
.into())
}
diff --git a/avro/src/serde/with.rs b/avro/src/serde/with.rs
index 29744e1..74c0a3a 100644
--- a/avro/src/serde/with.rs
+++ b/avro/src/serde/with.rs
@@ -503,6 +503,315 @@ pub mod slice_opt {
}
}
+/// (De)serialize [`BigDecimal`] as a [`Schema::BigDecimal`] instead of a [`Schema::String`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal_opt`] for optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal<'a> {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal")]
+/// decimal: BigDecimal,
+/// }
+/// ```
+///
+/// [`BigDecimal`]: ::bigdecimal::BigDecimal
+/// [`Schema::BigDecimal`]: crate::Schema::BigDecimal
+/// [`Schema::String`]: crate::Schema::String
+/// [`apache_avro::serde::bigdecimal_opt`]: bigdecimal_opt
+pub mod bigdecimal {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField},
+ serde::with::BytesType,
+ };
+
+ /// Returns [`Schema::BigDecimal`]
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::BigDecimal
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &BigDecimal, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&decimal_bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<BigDecimal, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ // We don't use &'de [u8] here as the deserializer doesn't support that
+ let bytes: Vec<u8> = serde_bytes::deserialize(deserializer)?;
+
+ deserialize_big_decimal(&bytes).map_err(D::Error::custom)
+ }
+}
+
+/// (De)serialize [`Option<BigDecimal>`] as a `Schema::Union(Schema::Null, Schema::BigDecimal)` instead of a `Schema::Union(Schema::Null, Schema::String)`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::bigdecimal`] for non-optional big decimals values.
+///
+/// When used with different serialization formats, this will write bytes.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBigDecimal<'a> {
+/// #[avro(with)]
+/// #[serde(with = "apache_avro::serde::bigdecimal_opt")]
+/// decimal: Option<BigDecimal>,
+/// }
+/// ```
+///
+/// [`Option<BigDecimal>`]: ::bigdecimal::BigDecimal
+/// [`apache_avro::serde::bigdecimal`]: bigdecimal
+pub mod bigdecimal_opt {
+ use std::collections::HashSet;
+
+ use bigdecimal::BigDecimal;
+ use serde::{Deserializer, Serializer, de::Error as _, ser::Error as _};
+
+ use crate::{
+ Schema,
+ bigdecimal::{big_decimal_as_bytes, deserialize_big_decimal},
+ schema::{Name, NamespaceRef, RecordField, UnionSchema},
+ serde::with::BytesType,
+ };
+
+ /// Returns `Schema::Union(Schema::Null, Schema::BigDecimal)`
+ pub fn get_schema_in_ctxt(_: &mut HashSet<Name>, _: NamespaceRef) -> Schema {
+ Schema::Union(
+ UnionSchema::new(vec![Schema::Null, Schema::BigDecimal])
+ .expect("This is a valid union"),
+ )
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<S>(decimal: &Option<BigDecimal>, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ let _guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ if let Some(decimal) = decimal {
+ let decimal_bytes = big_decimal_as_bytes(decimal).map_err(S::Error::custom)?;
+ serde_bytes::serialize(&Some(decimal_bytes), serializer)
+ } else {
+ serde_bytes::serialize(&None::<Vec<u8>>, serializer)
+ }
+ }
+
+ pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<BigDecimal>, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ let _bytes_guard = super::BytesTypeGuard::set(BytesType::Bytes);
+ let _guard = super::BorrowedGuard::set(true);
+ let bytes: Option<Vec<u8>> = serde_bytes::deserialize(deserializer)?;
+ if let Some(bytes) = bytes {
+ deserialize_big_decimal(&bytes)
+ .map(Some)
+ .map_err(D::Error::custom)
+ } else {
+ Ok(None)
+ }
+ }
+}
+
+/// (De)serialize an Rust array (`[T; N]`) as an Avro [`Schema::Array`].
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array_opt`] for optional array values.
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBytes<'a> {
+/// #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<i32>)]
+/// #[serde(with = "apache_avro::serde::array")]
+/// array: [i32; 10],
+/// }
+/// ```
+///
+/// [`apache_avro::serde::array_opt`]: array_opt
+/// [`Schema::Array`]: crate::schema::Schema::Array
+pub mod array {
+ use crate::{
+ AvroSchemaComponent, Schema,
+ schema::{Name, NamespaceRef, RecordField},
+ };
+ use serde::de::DeserializeOwned;
+ use serde::{Deserialize, Deserializer, Serialize, Serializer, de::Error as _};
+ use std::collections::HashSet;
+
+ /// Returns `Schema::Array(T::get_schema_in_ctxt())`
+ pub fn get_schema_in_ctxt<T: AvroSchemaComponent>(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build()
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<const N: usize, S, T>(value: &[T; N], serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ T: Serialize,
+ {
+ value.as_slice().serialize(serializer)
+ }
+
+ pub fn deserialize<'de, const N: usize, D, T>(deserializer: D) -> Result<[T; N], D::Error>
+ where
+ D: Deserializer<'de>,
+ T: DeserializeOwned,
+ {
+ let bytes = <Vec<T> as Deserialize>::deserialize(deserializer)?;
+ bytes.try_into().map_err(|v: Vec<T>| {
+ D::Error::custom(format!(
+ "Deserialized array has length {} which does not match array length of {N}",
+ v.len()
+ ))
+ })
+ }
+}
+
+/// (De)serialize an optional Rust array (`Option<[T; N]>`) as an Avro `Schema::Union([Schema::Null, Schema::Array])`.
+///
+/// This module is intended to be used through the Serde `with` attribute.
+///
+/// Use [`apache_avro::serde::array`] for non-optional array values.
+///
+/// When used with different serialization formats, this is equivalent to [`serde_bytes`].
+///
+/// See usage with below example:
+/// ```
+/// # use apache_avro::AvroSchema;
+/// # use serde::{Deserialize, Serialize};
+/// #[derive(AvroSchema, Serialize, Deserialize)]
+/// struct StructWithBytes<'a> {
+/// #[avro(with = apache_avro::serde::array_opt::get_schema_in_ctxt::<i32>)]
+/// #[serde(with = "apache_avro::serde::array_opt")]
+/// array: Option<[i32; 10]>,
+/// }
+/// ```
+///
+/// [`apache_avro::serde::array`]: mod@array
+pub mod array_opt {
+ use serde::{Deserialize, Deserializer, Serialize, Serializer, de::Error as _};
+ use std::collections::HashSet;
+
+ use crate::{
+ AvroSchemaComponent, Schema,
+ schema::{Name, NamespaceRef, RecordField, UnionSchema},
+ };
+
+ /// Returns `Schema::Union(Schema::Null, Schema::Array(T::get_schema_in_ctxt()))`
+ pub fn get_schema_in_ctxt<T: AvroSchemaComponent>(
+ named_schemas: &mut HashSet<Name>,
+ enclosing_namespace: NamespaceRef,
+ ) -> Schema {
+ Schema::Union(
+ UnionSchema::new(vec![
+ Schema::Null,
+ Schema::array(T::get_schema_in_ctxt(named_schemas, enclosing_namespace)).build(),
+ ])
+ .expect("This is a valid union"),
+ )
+ }
+
+ /// Returns `None`
+ pub fn get_record_fields_in_ctxt(
+ _: &mut HashSet<Name>,
+ _: NamespaceRef,
+ ) -> Option<Vec<RecordField>> {
+ None
+ }
+
+ pub fn serialize<const N: usize, S, T>(
+ value: &Option<[T; N]>,
+ serializer: S,
+ ) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ T: Serialize,
+ {
+ if let Some(array) = value {
+ Some(array.as_slice()).serialize(serializer)
+ } else {
+ None::<Vec<T>>.serialize(serializer)
+ }
+ }
+
+ pub fn deserialize<'de, const N: usize, D, T>(
+ deserializer: D,
+ ) -> Result<Option<[T; N]>, D::Error>
+ where
+ D: Deserializer<'de>,
+ T: Deserialize<'de>,
+ {
+ let bytes = <Option<Vec<T>> as Deserialize>::deserialize(deserializer)?;
+ if let Some(bytes) = bytes {
+ Ok(Some(bytes.try_into().map_err(|v: Vec<T>| {
+ D::Error::custom(format!(
+ "Deserialized array has length {} which does not match array length of {N}",
+ v.len()
+ ))
+ })?))
+ } else {
+ Ok(None)
+ }
+ }
+}
+
#[cfg(test)]
mod tests {
use crate::{Schema, from_value, to_value, types::Value};
diff --git a/avro/src/writer/mod.rs b/avro/src/writer/mod.rs
index 2d255a1..b11bbde 100644
--- a/avro/src/writer/mod.rs
+++ b/avro/src/writer/mod.rs
@@ -1160,7 +1160,7 @@ mod tests {
Err(e) => {
assert_eq!(
e.to_string(),
- r#"Failed to serialize field 'time' for record Record(RecordSchema { name: Name { name: "Conference", .. }, fields: [RecordField { name: "name", schema: String, .. }, RecordField { name: "date", aliases: ["time2", "time"], schema: Union(UnionSchema { schemas: [Null, Long] }), .. }], .. }): Failed to serialize value of type f64 using schema Union(UnionSchema { schemas: [Null, Long] }): 12345678.9. Cause: Cannot find a Double schema in [Null, Long]"#
+ r#"Failed to serialize field 'date' of record RecordSchema { name: Name { name: "Conference", .. }, fields: [RecordField { name: "name", schema: String, .. }, RecordField { name: "date", aliases: ["time2", "time"], schema: Union(UnionSchema { schemas: [Null, Long] }), .. }], .. }: Failed to serialize value of type `f64`: Expected Schema::Double"#
);
}
}
diff --git a/avro/tests/serde_human_readable_false.rs b/avro/tests/serde_human_readable_false.rs
index 2f8be3f..544e09e 100644
--- a/avro/tests/serde_human_readable_false.rs
+++ b/avro/tests/serde_human_readable_false.rs
@@ -79,7 +79,7 @@ fn avro_rs_440_uuid_string() -> TestResult {
let writer = SpecificSingleObjectWriter::new()?;
assert_eq!(
writer.write(uuid, &mut buffer).unwrap_err().to_string(),
- "Failed to serialize value of type bytes using schema Uuid(String): 55e840e29b41d4a7164466554400. Cause: Expected a string, but got 16 bytes. Did you mean to use `Schema::Uuid(UuidSchema::Fixed)` or `utils::serde_set_human_readable(true)`?"
+ "Failed to serialize value of type `bytes` using Schema::Uuid(String): Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Bytes | Fixed) | Schema::Duration"
);
Ok(())
diff --git a/avro/tests/serde_human_readable_true.rs b/avro/tests/serde_human_readable_true.rs
index d1c71d7..c0612bb 100644
--- a/avro/tests/serde_human_readable_true.rs
+++ b/avro/tests/serde_human_readable_true.rs
@@ -103,7 +103,7 @@ fn avro_rs_440_uuid_bytes() -> TestResult {
let writer = SpecificSingleObjectWriter::new()?;
assert_eq!(
writer.write(uuid, &mut buffer).unwrap_err().to_string(),
- r#"Failed to serialize value of type string using schema Uuid(Bytes): 550e8400-e29b-41d4-a716-446655440000. Cause: Expected bytes but got a string. Did you mean to use `Schema::Uuid(UuidSchema::String)` or `utils::serde_set_human_readable(false)`?"#
+ r#"Failed to serialize value of type `str` using Schema::Uuid(Bytes): Expected Schema::String | Schema::Uuid(String)"#
);
Ok(())
@@ -129,7 +129,7 @@ fn avro_rs_440_uuid_fixed() -> TestResult {
let writer = SpecificSingleObjectWriter::new()?;
assert_eq!(
writer.write(uuid, &mut buffer).unwrap_err().to_string(),
- r#"Failed to serialize value of type string using schema Uuid(Fixed(FixedSchema { name: Name { name: "uuid", .. }, size: 16, .. })): 550e8400-e29b-41d4-a716-446655440000. Cause: Expected bytes but got a string. Did you mean to use `Schema::Uuid(UuidSchema::String)` or `utils::serde_set_human_readable(false)`?"#
+ r#"Failed to serialize value of type `str` using Schema::Uuid(Fixed(FixedSchema { name: Name { name: "uuid", .. }, size: 16, .. })): Expected Schema::String | Schema::Uuid(String)"#
);
Ok(())
diff --git a/avro_derive/tests/derive.proptest-regressions b/avro_derive/tests/derive.proptest-regressions
new file mode 100644
index 0000000..d9ec339
--- /dev/null
+++ b/avro_derive/tests/derive.proptest-regressions
@@ -0,0 +1,8 @@
+# Seeds for failure cases proptest has generated in the past. It is
+# automatically read and these particular cases re-run before any
+# novel cases are generated.
+#
+# It is recommended to check this file in to source control so that
+# everyone who runs the test benefits from these saved cases.
+cc fcee0d6e798e3fea29e4567deeaa845a8868672476574c6f1c6723734e8852b7 # shrinks to a = [], b = [0, 0]
+cc 3b35e254022cadeafda5adb27919361b91e7b77a07ed1eea7e0919856aa247d0 # shrinks to a = "", b = [], c = {"": 0}
diff --git a/avro_derive/tests/derive.rs b/avro_derive/tests/derive.rs
index ae366e3..66d179e 100644
--- a/avro_derive/tests/derive.rs
+++ b/avro_derive/tests/derive.rs
@@ -33,6 +33,7 @@ use apache_avro::schema::NamespaceRef;
use pretty_assertions::assert_eq;
/// Takes in a type that implements the right combination of traits and runs it through a Serde Cycle and asserts the result is the same
+#[track_caller]
fn serde_assert<T>(obj: T)
where
T: std::fmt::Debug + Serialize + DeserializeOwned + AvroSchema + Clone + PartialEq,
@@ -40,6 +41,7 @@ where
assert_eq!(obj, serde(obj.clone()));
}
+#[track_caller]
fn serde<T>(obj: T) -> T
where
T: Serialize + DeserializeOwned + AvroSchema,
@@ -47,6 +49,7 @@ where
de(ser(obj))
}
+#[track_caller]
fn ser<T>(obj: T) -> Vec<u8>
where
T: Serialize + AvroSchema,
@@ -59,6 +62,7 @@ where
writer.into_inner().unwrap()
}
+#[track_caller]
fn de<T>(encoded: Vec<u8>) -> T
where
T: DeserializeOwned + AvroSchema,
@@ -800,6 +804,8 @@ fn test_cons_generic() {
#[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq, Eq)]
struct TestSimpleArray {
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<i32>)]
+ #[serde(with = "apache_avro::serde::array")]
a: [i32; 4],
}
@@ -828,7 +834,9 @@ fn test_simple_array(a: [i32; 4]) {
}}
#[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
-struct TestComplexArray<T: AvroSchemaComponent> {
+struct TestComplexArray<T: AvroSchemaComponent + Serialize + DeserializeOwned> {
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<T>)]
+ #[serde(with = "apache_avro::serde::array")]
a: [T; 2],
}
@@ -882,6 +890,8 @@ fn test_complex_array() {
#[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq, Eq)]
struct Testu8 {
a: Vec<u8>,
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<u8>)]
+ #[serde(with = "apache_avro::serde::array")]
b: [u8; 2],
}
@@ -2063,6 +2073,8 @@ fn avro_rs_401_supported_type_variants() {
three: &'static i32,
four: &'a str,
five: &'a mut f64,
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<u8>)]
+ #[serde(with = "apache_avro::serde::array")]
six: [u8; 5],
seven: [u8],
}
@@ -2390,7 +2402,11 @@ fn avro_rs_476_field_default() {
_m: char,
_n: Box<Spam>,
_o: Vec<bool>,
+ #[serde(with = "apache_avro::serde::array")]
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<u8>)]
_p: [u8; 5],
+ #[serde(with = "apache_avro::serde::array")]
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<Bar>)]
_p_alt: [Bar; 5],
_q: HashMap<String, String>,
_r: Option<f64>,
@@ -2406,7 +2422,7 @@ fn avro_rs_476_field_default() {
let schema = Foo::get_schema();
assert_eq!(
serde_json::to_string(&schema).unwrap(),
- r#"{"type":"record","name":"Foo","fields":[{"name":"_a","type":"boolean"},{"name":"_b","type":"int"},{"name":"_c","type":"int"},{"name":"_d","type":"int"},{"name":"_e","type":"long"},{"name":"_f","type":"int"},{"name":"_g","type":"int"},{"name":"_h","type":"long"},{"name":"_i","type":"float"},{"name":"_j","type":"double"},{"name":"_k","type":"string"},{"name":"_l","type":"string"},{"name":"_m","type":"string"},{"name":"_n","type":{"type":"record","name":"Spam","fields":[{"name":" [...]
+ r#"{"type":"record","name":"Foo","fields":[{"name":"_a","type":"boolean"},{"name":"_b","type":"int"},{"name":"_c","type":"int"},{"name":"_d","type":"int"},{"name":"_e","type":"long"},{"name":"_f","type":"int"},{"name":"_g","type":"int"},{"name":"_h","type":"long"},{"name":"_i","type":"float"},{"name":"_j","type":"double"},{"name":"_k","type":"string"},{"name":"_l","type":"string"},{"name":"_m","type":"string"},{"name":"_n","type":{"type":"record","name":"Spam","fields":[{"name":" [...]
);
}
@@ -2479,8 +2495,11 @@ fn avro_rs_476_field_default_provided() {
_n: Box<Spam>,
#[avro(default = "[true, false, true]")]
_o: Vec<bool>,
- #[avro(default = "[1,2,3,4,5]")]
+ #[serde(with = "apache_avro::serde::array")]
+ #[avro(default = "[1,2,3,4,5]", with = apache_avro::serde::array::get_schema_in_ctxt::<u8>)]
_p: [u8; 5],
+ #[avro(with = apache_avro::serde::array::get_schema_in_ctxt::<Spam>)]
+ #[serde(with = "apache_avro::serde::array")]
#[avro(
default = r#"[{"_field": true},{"_field": false},{"_field": true},{"_field": false},{"_field": true}]"#
)]
@@ -2514,7 +2533,7 @@ fn avro_rs_476_field_default_provided() {
let schema = Foo::get_schema();
assert_eq!(
serde_json::to_string(&schema).unwrap(),
- r#"{"type":"record","name":"Foo","fields":[{"name":"_a","type":"boolean","default":true},{"name":"_b","type":"int","default":42},{"name":"_c","type":"int","default":42},{"name":"_d","type":"int","default":42},{"name":"_e","type":"long","default":42},{"name":"_f","type":"int","default":42},{"name":"_g","type":"int","default":42},{"name":"_h","type":"long","default":42},{"name":"_i","type":"float","default":42.0},{"name":"_j","type":"double","default":42.0},{"name":"_k","type":"str [...]
+ r#"{"type":"record","name":"Foo","fields":[{"name":"_a","type":"boolean","default":true},{"name":"_b","type":"int","default":42},{"name":"_c","type":"int","default":42},{"name":"_d","type":"int","default":42},{"name":"_e","type":"long","default":42},{"name":"_f","type":"int","default":42},{"name":"_g","type":"int","default":42},{"name":"_h","type":"long","default":42},{"name":"_i","type":"float","default":42.0},{"name":"_j","type":"double","default":42.0},{"name":"_k","type":"str [...]
);
}
diff --git a/avro_derive/tests/serde.rs b/avro_derive/tests/serde.rs
index cf11fa4..448d59f 100644
--- a/avro_derive/tests/serde.rs
+++ b/avro_derive/tests/serde.rs
@@ -20,6 +20,7 @@ use serde::{Deserialize, Serialize, de::DeserializeOwned};
/// Takes in a type that implements the right combination of traits and runs it through a Serde
/// round-trip and asserts the result is the same.
+#[track_caller]
fn serde_assert<T>(obj: T)
where
T: std::fmt::Debug + Serialize + DeserializeOwned + AvroSchema + Clone + PartialEq,
@@ -29,15 +30,13 @@ where
/// Takes in a type that implements the right combination of traits and runs it through a Serde
/// round-trip and asserts that the error matches the expected string.
+#[track_caller]
fn serde_assert_err<T>(obj: T, expected: &str)
where
T: std::fmt::Debug + Serialize + DeserializeOwned + AvroSchema + Clone + PartialEq,
{
let error = serde(obj).unwrap_err().to_string();
- assert!(
- error.contains(expected),
- "Error `{error}` does not contain `{expected}`"
- );
+ assert_eq!(error, expected);
}
fn serde<T>(obj: T) -> Result<T, Error>
@@ -313,7 +312,7 @@ mod container_attributes {
a: "spam".to_string(),
b: 321,
},
- "Invalid field name c",
+ r#"Missing field in record: "c""#,
);
}
(avro-rs) 02/05: feat: `SchemaAwareDeserializer` (no tests)
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a commit to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
commit 764f119de4a67de855dffe5883eae1bcdc58953e
Author: Kriskras99 <gi...@kriskras99.nl>
AuthorDate: Fri Mar 13 18:50:41 2026 +0100
feat: `SchemaAwareDeserializer` (no tests)
---
avro/src/error.rs | 11 +-
avro/src/schema/union.rs | 13 +-
avro/src/serde/deser_schema/array.rs | 99 ++++
avro/src/serde/deser_schema/enums.rs | 233 ++++++++++
avro/src/serde/deser_schema/identifier.rs | 269 +++++++++++
avro/src/serde/deser_schema/map.rs | 147 ++++++
avro/src/serde/deser_schema/mod.rs | 741 ++++++++++++++++++++++++++++++
avro/src/serde/deser_schema/record.rs | 93 ++++
avro/src/serde/deser_schema/tuple.rs | 123 +++++
avro/src/serde/mod.rs | 1 +
10 files changed, 1728 insertions(+), 2 deletions(-)
diff --git a/avro/src/error.rs b/avro/src/error.rs
index 67ff13f..123bf3c 100644
--- a/avro/src/error.rs
+++ b/avro/src/error.rs
@@ -156,7 +156,6 @@ pub enum Details {
#[error("Union index {index} out of bounds: {num_variants}")]
GetUnionVariant { index: i64, num_variants: usize },
- #[deprecated(since = "0.20.0", note = "This error variant is not generated anymore")]
#[error("Enum symbol index out of bounds: {num_variants}")]
EnumSymbolIndex { index: usize, num_variants: usize },
@@ -579,6 +578,16 @@ pub enum Details {
#[error("Failed to deserialize Avro value into value: {0}")]
DeserializeValue(String),
+ #[error("Failed to deserialize value of type {value_type} using schema {schema:?}: {value}")]
+ DeserializeSchemaAware {
+ value_type: &'static str,
+ value: String,
+ schema: Schema,
+ },
+
+ #[error("Only expected `deserialize_identifier` to be called but `{0}` was called")]
+ DeserializeIdentifier(&'static str),
+
#[error("Failed to write buffer bytes during flush: {0}")]
WriteBytes(#[source] std::io::Error),
diff --git a/avro/src/schema/union.rs b/avro/src/schema/union.rs
index 31affe7..c8bef75 100644
--- a/avro/src/schema/union.rs
+++ b/avro/src/schema/union.rs
@@ -74,7 +74,18 @@ impl UnionSchema {
&self.schemas
}
- /// Returns true if the any of the variants of this `UnionSchema` is `Null`.
+ /// Get the variant at the given index.
+ pub fn get_variant(&self, index: usize) -> Result<&Schema, Error> {
+ self.schemas.get(index).ok_or_else(|| {
+ Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.schemas.len(),
+ }
+ .into()
+ })
+ }
+
+ /// Returns true if any of the variants of this `UnionSchema` is `Null`.
pub fn is_nullable(&self) -> bool {
self.variant_index.contains_key(&SchemaKind::Null)
}
diff --git a/avro/src/serde/deser_schema/array.rs b/avro/src/serde/deser_schema/array.rs
new file mode 100644
index 0000000..c1da742
--- /dev/null
+++ b/avro/src/serde/deser_schema/array.rs
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{Config, SchemaAwareDeserializer};
+use crate::schema::ArraySchema;
+use crate::util::{zag_i32, zag_i64};
+use crate::{Error, Schema};
+use serde::de::{DeserializeSeed, SeqAccess};
+use std::borrow::Borrow;
+use std::io::Read;
+
+/// Deserialize sequences from an Avro array.
+pub struct ArrayDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ /// Track where we are in reading the array
+ remaining: Option<u32>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> ArrayDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+ let remaining = Self::read_block_header(reader)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ remaining,
+ })
+ }
+
+ fn read_block_header(reader: &mut R) -> Result<Option<u32>, Error> {
+ let remaining = zag_i32(reader)?;
+ if remaining < 0 {
+ // If the block size is negative the next number is the size of the block in bytes
+ let _bytes = zag_i64(reader)?;
+ }
+ if remaining == 0 {
+ // If the block size is zero the array is finished
+ Ok(None)
+ } else {
+ Ok(Some(remaining.unsigned_abs()))
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de> for ArrayDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if let Some(mut remaining) = self.remaining {
+ let value = seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)?;
+ remaining -= 1;
+ if remaining == 0 {
+ self.remaining = Self::read_block_header(self.reader)?;
+ } else {
+ self.remaining = Some(remaining);
+ }
+ Ok(Some(value))
+ } else {
+ Ok(None)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ self.remaining
+ .map(|x| usize::try_from(x).unwrap_or(usize::MAX))
+ }
+}
diff --git a/avro/src/serde/deser_schema/enums.rs b/avro/src/serde/deser_schema/enums.rs
new file mode 100644
index 0000000..bb2e7c5
--- /dev/null
+++ b/avro/src/serde/deser_schema/enums.rs
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{Config, DESERIALIZE_ANY, SchemaAwareDeserializer, identifier::IdentifierDeserializer};
+use crate::error::Details;
+use crate::schema::{EnumSchema, UnionSchema};
+use crate::util::zag_i32;
+use crate::{Error, Schema};
+use serde::Deserializer;
+use serde::de::{DeserializeSeed, EnumAccess, Unexpected, VariantAccess, Visitor};
+use std::borrow::Borrow;
+use std::io::Read;
+
+/// Deserializer for plain enums.
+pub struct PlainEnumDeserializer<'s, 'r, R: Read> {
+ reader: &'r mut R,
+ symbols: &'s [String],
+}
+
+impl<'s, 'r, R: Read> PlainEnumDeserializer<'s, 'r, R> {
+ pub fn new(reader: &'r mut R, schema: &'s EnumSchema) -> Self {
+ Self {
+ symbols: &schema.symbols,
+ reader,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read> EnumAccess<'de> for PlainEnumDeserializer<'s, 'r, R> {
+ type Error = Error;
+ type Variant = Self;
+
+ fn variant_seed<V>(self, seed: V) -> Result<(V::Value, Self::Variant), Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let symbol = self.symbols.get(index).ok_or(Details::EnumSymbolIndex {
+ index,
+ num_variants: self.symbols.len(),
+ })?;
+ Ok((
+ seed.deserialize(IdentifierDeserializer::string(symbol))?,
+ self,
+ ))
+ }
+}
+
+impl<'de, 's, 'r, R: Read> VariantAccess<'de> for PlainEnumDeserializer<'s, 'r, R> {
+ type Error = Error;
+
+ fn unit_variant(self) -> Result<(), Self::Error> {
+ Ok(())
+ }
+
+ fn newtype_variant_seed<T>(self, _: T) -> Result<T::Value, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ let unexp = Unexpected::UnitVariant;
+ Err(serde::de::Error::invalid_type(unexp, &"newtype variant"))
+ }
+
+ fn tuple_variant<V>(self, _: usize, _: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ let unexp = Unexpected::UnitVariant;
+ Err(serde::de::Error::invalid_type(unexp, &"tuple variant"))
+ }
+
+ fn struct_variant<V>(self, _: &'static [&'static str], _: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ let unexp = Unexpected::UnitVariant;
+ Err(serde::de::Error::invalid_type(unexp, &"struct variant"))
+ }
+}
+
+pub struct UnionEnumDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ variants: &'s [Schema],
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> UnionEnumDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ variants: schema.variants(),
+ config,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> EnumAccess<'de>
+ for UnionEnumDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+ type Variant = UnionVariantAccess<'s, 'r, R, S>;
+
+ fn variant_seed<V>(self, seed: V) -> Result<(V::Value, Self::Variant), Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let index = zag_i32(self.reader)?;
+ let index = usize::try_from(index).map_err(|e| Details::ConvertI32ToUsize(e, index))?;
+ let schema = self.variants.get(index).ok_or(Details::GetUnionVariant {
+ index: index as i64,
+ num_variants: self.variants.len(),
+ })?;
+
+ Ok((
+ seed.deserialize(IdentifierDeserializer::index(index as u32))?,
+ UnionVariantAccess::new(schema, self.reader, self.config)?,
+ ))
+ }
+}
+
+pub struct UnionVariantAccess<'s, 'r, R: Read, S: Borrow<Schema>> {
+ schema: &'s Schema,
+ reader: &'r mut R,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> UnionVariantAccess<'s, 'r, R, S> {
+ pub fn new(
+ schema: &'s Schema,
+ reader: &'r mut R,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema {
+ config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(Self {
+ schema,
+ reader,
+ config,
+ })
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> VariantAccess<'de>
+ for UnionVariantAccess<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn unit_variant(self) -> Result<(), Self::Error> {
+ match self.schema {
+ Schema::Null => Ok(()),
+ Schema::Record(record) if record.fields.is_empty() => Ok(()),
+ _ => Err(self.error(
+ "unit variant",
+ "Expected Schema::Null | Schema::Record(fields.len() == 0)",
+ )),
+ }
+ }
+
+ fn newtype_variant_seed<T>(self, seed: T) -> Result<T::Value, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ match self.schema {
+ Schema::Record(record)
+ if record.fields.len() == 1
+ && record
+ .attributes
+ .get("org.apache.avro.rust.union_of_records")
+ == Some(&serde_json::Value::Bool(true)) =>
+ {
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ &record.fields[0].schema,
+ self.config,
+ )?)
+ }
+ _ => seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?),
+ }
+ }
+
+ fn tuple_variant<V>(self, len: usize, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ SchemaAwareDeserializer::new(self.reader, self.schema, self.config)?
+ .deserialize_tuple(len, visitor)
+ }
+
+ fn struct_variant<V>(
+ self,
+ fields: &'static [&'static str],
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ SchemaAwareDeserializer::new(self.reader, self.schema, self.config)?.deserialize_struct(
+ DESERIALIZE_ANY,
+ fields,
+ visitor,
+ )
+ }
+}
diff --git a/avro/src/serde/deser_schema/identifier.rs b/avro/src/serde/deser_schema/identifier.rs
new file mode 100644
index 0000000..de51f1a
--- /dev/null
+++ b/avro/src/serde/deser_schema/identifier.rs
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::Error;
+use crate::error::Details;
+use serde::Deserializer;
+use serde::de::Visitor;
+
+/// Deserializer that only accepts `deserialize_identifier` calls.
+pub enum IdentifierDeserializer<'s> {
+ String(&'s str),
+ Index(u32),
+}
+
+impl<'s> IdentifierDeserializer<'s> {
+ pub fn string(name: &'s str) -> Self {
+ Self::String(name)
+ }
+
+ pub fn index(index: u32) -> Self {
+ Self::Index(index)
+ }
+
+ fn error(&self, error: &'static str) -> Error {
+ Error::new(Details::DeserializeIdentifier(error))
+ }
+}
+
+impl<'de, 's> Deserializer<'de> for IdentifierDeserializer<'s> {
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_identifier(visitor)
+ }
+
+ fn deserialize_bool<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_bool"))
+ }
+
+ fn deserialize_i8<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_i8"))
+ }
+
+ fn deserialize_i16<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_i16"))
+ }
+ fn deserialize_i32<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_i32"))
+ }
+
+ fn deserialize_i64<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_i64"))
+ }
+
+ fn deserialize_u8<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_u8"))
+ }
+
+ fn deserialize_u16<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_u16"))
+ }
+ fn deserialize_u32<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_u32"))
+ }
+
+ fn deserialize_u64<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_u64"))
+ }
+
+ fn deserialize_f32<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_f32"))
+ }
+
+ fn deserialize_f64<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_f64"))
+ }
+
+ fn deserialize_char<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_char"))
+ }
+
+ fn deserialize_str<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_str"))
+ }
+
+ fn deserialize_string<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_string"))
+ }
+
+ fn deserialize_bytes<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_bytes"))
+ }
+
+ fn deserialize_byte_buf<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_byte_buf"))
+ }
+ fn deserialize_option<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_option"))
+ }
+
+ fn deserialize_unit<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_unit"))
+ }
+
+ fn deserialize_unit_struct<V>(
+ self,
+ _name: &'static str,
+ _visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_unit_struct"))
+ }
+ fn deserialize_newtype_struct<V>(
+ self,
+ _name: &'static str,
+ _visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_newtype_struct"))
+ }
+
+ fn deserialize_seq<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_seq"))
+ }
+
+ fn deserialize_tuple<V>(self, _len: usize, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_tuple"))
+ }
+
+ fn deserialize_tuple_struct<V>(
+ self,
+ _name: &'static str,
+ _len: usize,
+ _visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_tuple_struct"))
+ }
+
+ fn deserialize_map<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_map"))
+ }
+
+ fn deserialize_struct<V>(
+ self,
+ _name: &'static str,
+ _fields: &'static [&'static str],
+ _visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_struct"))
+ }
+ fn deserialize_enum<V>(
+ self,
+ _name: &'static str,
+ _variants: &'static [&'static str],
+ _visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_enum"))
+ }
+
+ fn deserialize_identifier<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self {
+ IdentifierDeserializer::String(string) => visitor.visit_str(string),
+ IdentifierDeserializer::Index(index) => visitor.visit_u32(index),
+ }
+ }
+
+ fn deserialize_ignored_any<V>(self, _visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ Err(self.error("deserialize_ignored_any"))
+ }
+}
diff --git a/avro/src/serde/deser_schema/map.rs b/avro/src/serde/deser_schema/map.rs
new file mode 100644
index 0000000..90508da
--- /dev/null
+++ b/avro/src/serde/deser_schema/map.rs
@@ -0,0 +1,147 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::schema::MapSchema;
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
+use crate::util::{zag_i32, zag_i64};
+use crate::{Error, Schema};
+use serde::de::{DeserializeSeed, MapAccess};
+use std::borrow::Borrow;
+use std::io::Read;
+
+pub struct MapDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ remaining: Option<u32>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> MapDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+ let remaining = Self::read_block_header(reader)?;
+ Ok(Self {
+ schema,
+ reader,
+ config,
+ remaining,
+ })
+ }
+
+ fn read_block_header(reader: &mut R) -> Result<Option<u32>, Error> {
+ let remaining = zag_i32(reader)?;
+ if remaining < 0 {
+ // If the block size is negative the next number is the size of the block in bytes
+ let _bytes = zag_i64(reader)?;
+ }
+ if remaining == 0 {
+ // If the block size is zero the array is finished
+ Ok(None)
+ } else {
+ Ok(Some(remaining.unsigned_abs()))
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> MapAccess<'de> for MapDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_key_seed<K>(&mut self, seed: K) -> Result<Option<K::Value>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ {
+ if self.remaining.is_some() {
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ &Schema::String,
+ self.config,
+ )?)
+ .map(Some)
+ } else {
+ Ok(None)
+ }
+ }
+
+ fn next_value_seed<V>(&mut self, seed: V) -> Result<V::Value, Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let mut remaining = self
+ .remaining
+ .expect("next_key returned None, next_value should not have been called");
+ let value = seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)?;
+
+ remaining -= 1;
+ if remaining == 0 {
+ self.remaining = Self::read_block_header(self.reader)?;
+ } else {
+ self.remaining = Some(remaining);
+ }
+ Ok(value)
+ }
+
+ fn next_entry_seed<K, V>(
+ &mut self,
+ kseed: K,
+ vseed: V,
+ ) -> Result<Option<(K::Value, V::Value)>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ V: DeserializeSeed<'de>,
+ {
+ if let Some(mut remaining) = self.remaining {
+ let key = kseed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ &Schema::String,
+ self.config,
+ )?)?;
+ let value = vseed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)?;
+
+ remaining -= 1;
+ if remaining == 0 {
+ self.remaining = Self::read_block_header(self.reader)?;
+ } else {
+ self.remaining = Some(remaining);
+ }
+
+ Ok(Some((key, value)))
+ } else {
+ Ok(None)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ self.remaining
+ .map(|x| usize::try_from(x).unwrap_or(usize::MAX))
+ }
+}
diff --git a/avro/src/serde/deser_schema/mod.rs b/avro/src/serde/deser_schema/mod.rs
new file mode 100644
index 0000000..99d5897
--- /dev/null
+++ b/avro/src/serde/deser_schema/mod.rs
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::decode::decode_len;
+use crate::error::Details;
+use crate::schema::{DecimalSchema, InnerDecimalSchema, Name, UnionSchema, UuidSchema};
+use crate::util::{zag_i32, zag_i64};
+use crate::{Error, Schema};
+use serde::de::{Deserializer, Visitor};
+use std::borrow::Borrow;
+use std::collections::HashMap;
+use std::io::Read;
+
+mod array;
+mod enums;
+mod identifier;
+mod map;
+mod record;
+mod tuple;
+
+use crate::serde::deser_schema::enums::UnionEnumDeserializer;
+use array::ArrayDeserializer;
+use enums::PlainEnumDeserializer;
+use map::MapDeserializer;
+use record::RecordDeserializer;
+use tuple::{ManyTupleDeserializer, OneTupleDeserializer};
+
+/// Configure the deserializer.
+#[derive(Debug)]
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
+ ///
+ /// This map is not allowed to contain any [`Schema::Ref`], the deserializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// Was the data serialized with `human_readable`.
+ pub human_readable: bool,
+}
+
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
+ }
+}
+
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+/// A deserializer that deserializes directly from raw Avro datum.
+pub struct SchemaAwareDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ /// The schema of the data being deserialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> SchemaAwareDeserializer<'s, 'r, R, S> {
+ /// Create a new deserializer for this schema.
+ ///
+ /// This will resolve a [`Schema::Ref`] to its actual schema.
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ if let Schema::Ref { name } = schema {
+ let schema = config.get_schema(name)?;
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ } else {
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ })
+ }
+ }
+
+ /// Create an error for the current type being deserialized with the given message.
+ ///
+ /// This will also include the current schema.
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::DeserializeSchemaAware {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
+ }
+
+ /// Create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(self)
+ }
+
+ /// Read the union and create a new deserializer with the existing reader and config.
+ ///
+ /// This will resolve the read schema if it is a reference.
+ fn with_union(self, schema: &'s UnionSchema) -> Result<Self, Error> {
+ let index = zag_i32(self.reader)?;
+ let variant = schema.get_variant(index as usize)?;
+ self.with_different_schema(variant)
+ }
+
+ /// Read an integer from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_int(&mut self, original_ty: &'static str) -> Result<i32, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zag_i32(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
+ )),
+ }
+ }
+
+ /// Read a long from the reader.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// It does not read [`Schema::Union`]s.
+ fn checked_read_long(&mut self, original_ty: &'static str) -> Result<i64, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => zag_i64(self.reader),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}",
+ )),
+
+ }
+ }
+
+ /// Read a string from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_string(&mut self) -> Result<String, Error> {
+ let bytes = self.read_bytes_with_len()?;
+ Ok(String::from_utf8(bytes).map_err(Details::ConvertToUtf8)?)
+ }
+
+ /// Read a bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes_with_len(&mut self) -> Result<Vec<u8>, Error> {
+ let length = decode_len(self.reader)?;
+ self.read_bytes(length)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_bytes(&mut self, length: usize) -> Result<Vec<u8>, Error> {
+ let mut buf = vec![0; length];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+
+ /// Read `n` bytes from the reader.
+ ///
+ /// This does not check the current schema.
+ fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error> {
+ let mut buf = [0; N];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ Ok(buf)
+ }
+}
+
+/// A static string that will bypass name checks in `deserialize_*` functions.
+///
+/// This is used so that `deserialize_any` can use the `deserialize_*` implementation which take
+/// a static string.
+///
+/// We don't want users to abuse this feature so this value is compared by pointer address, therefore
+/// a user providing the string below will not be able to skip name validation.
+static DESERIALIZE_ANY: &str = "This value is compared by pointer value";
+/// A static array so that `deserialize_any` can call `deserialize_*` functions.
+static DESERIALIZE_ANY_FIELDS: &[&str] = &[];
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> Deserializer<'de>
+ for SchemaAwareDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => self.deserialize_unit(visitor),
+ Schema::Boolean => self.deserialize_bool(visitor),
+ Schema::Int | Schema::Date | Schema::TimeMillis => self.deserialize_i32(visitor),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => self.deserialize_i64(visitor),
+ Schema::Float => self.deserialize_f32(visitor),
+ Schema::Double => self.deserialize_f64(visitor),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Decimal(_)
+ | Schema::BigDecimal
+ | Schema::Uuid(UuidSchema::Fixed(_) | UuidSchema::Bytes)
+ | Schema::Duration(_) => self.deserialize_byte_buf(visitor),
+ Schema::String | Schema::Uuid(UuidSchema::String) => self.deserialize_string(visitor),
+ Schema::Array(_) => self.deserialize_seq(visitor),
+ Schema::Map(_) => self.deserialize_map(visitor),
+ Schema::Union(union) => self.with_union(union)?.deserialize_any(visitor),
+ Schema::Record(schema) => {
+ if schema.attributes.get("org.apache.avro.rust.tuple")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ // This attribute is needed because we can't tell the difference between a tuple
+ // and struct, but a tuple needs to be deserialized as a sequence instead of a map.
+ self.deserialize_tuple(schema.fields.len(), visitor)
+ } else {
+ self.deserialize_struct(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ }
+ Schema::Enum(_) => {
+ self.deserialize_enum(DESERIALIZE_ANY, DESERIALIZE_ANY_FIELDS, visitor)
+ }
+ Schema::Ref { .. } => unreachable!("References are resolved on deserializer creation"),
+ }
+ }
+
+ fn deserialize_bool<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Boolean => {
+ let mut buf = [0xFF];
+ self.reader
+ .read_exact(&mut buf)
+ .map_err(Details::ReadBytes)?;
+ match buf[0] {
+ 0 => visitor.visit_bool(false),
+ 1 => visitor.visit_bool(true),
+ _ => Err(self.error("bool", format!("{} is not a valid boolean", buf[0]))),
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_bool(visitor),
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
+ }
+
+ fn deserialize_i8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i8(visitor)
+ } else {
+ let int = self.checked_read_int("i8")?;
+ let value = i8::try_from(int)
+ .map_err(|_| self.error("i8", format!("Could not convert int ({int}) to an i8")))?;
+ visitor.visit_i8(value)
+ }
+ }
+
+ fn deserialize_i16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i16(visitor)
+ } else {
+ let int = self.checked_read_int("i16")?;
+ let value = i16::try_from(int).map_err(|_| {
+ self.error("i16", format!("Could not convert int ({int}) to an i16"))
+ })?;
+ visitor.visit_i16(value)
+ }
+ }
+
+ fn deserialize_i32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i32(visitor)
+ } else {
+ visitor.visit_i32(self.checked_read_int("i32")?)
+ }
+ }
+
+ fn deserialize_i64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_i64(visitor)
+ } else {
+ visitor.visit_i64(self.checked_read_long("i64")?)
+ }
+ }
+
+ fn deserialize_i128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ visitor.visit_i128(i128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_u8<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u8(visitor)
+ } else {
+ let int = self.checked_read_int("u8")?;
+ let value = u8::try_from(int)
+ .map_err(|_| self.error("u8", format!("Could not convert int ({int}) to an u8")))?;
+ visitor.visit_u8(value)
+ }
+ }
+
+ fn deserialize_u16<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u16(visitor)
+ } else {
+ let int = self.checked_read_int("u16")?;
+ let value = u16::try_from(int).map_err(|_| {
+ self.error("u16", format!("Could not convert int ({int}) to an u16"))
+ })?;
+ visitor.visit_u16(value)
+ }
+ }
+
+ fn deserialize_u32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema {
+ self.with_union(union)?.deserialize_u32(visitor)
+ } else {
+ let long = self.checked_read_long("u32")?;
+ let value = u32::try_from(long).map_err(|_| {
+ self.error("u32", format!("Could not convert long ({long}) to an u32"))
+ })?;
+ visitor.visit_u32(value)
+ }
+ }
+
+ fn deserialize_u64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ visitor.visit_u64(u64::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_u64(visitor),
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
+ }
+
+ fn deserialize_u128<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ visitor.visit_u128(u128::from_le_bytes(self.read_array()?))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_i128(visitor),
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
+ }
+
+ fn deserialize_f32<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Float => visitor.visit_f32(f32::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f32(visitor),
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
+ }
+
+ fn deserialize_f64<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Double => visitor.visit_f64(f64::from_le_bytes(self.read_array()?)),
+ Schema::Union(union) => self.with_union(union)?.deserialize_f64(visitor),
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
+ }
+
+ fn deserialize_char<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // A char cannot be deserialized using Schema::Uuid(UuidSchema::String) as that is at least
+ // 32 characters.
+ Schema::String => {
+ let string = self.read_string()?;
+ let mut chars = string.chars();
+ let char = chars
+ .next()
+ .ok_or_else(|| self.error("char", "String is empty"))?;
+ // We can't just check the string length, as that is in bytes not characters
+ if chars.next().is_some() {
+ Err(self.error(
+ "char",
+ format!(r#"Read more than one character: "{string}""#),
+ ))
+ } else {
+ visitor.visit_char(char)
+ }
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_char(visitor),
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_str<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_string(visitor)
+ }
+
+ fn deserialize_string<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ visitor.visit_string(self.read_string()?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_string(visitor),
+ _ => Err(self.error("string", "Expected Schema::String | Schema::Uuid(String)")),
+ }
+ }
+
+ fn deserialize_bytes<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ self.deserialize_byte_buf(visitor)
+ }
+
+ fn deserialize_byte_buf<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ visitor.visit_byte_buf(self.read_bytes_with_len()?)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), ..}) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ visitor.visit_byte_buf(self.read_bytes(fixed.size)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_byte_buf(visitor),
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Fixed | Bytes) | Schema::Duration")),
+ }
+ }
+
+ fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && union.is_nullable()
+ {
+ let index = zag_i32(self.reader)?;
+ let schema = union.get_variant(index as usize)?;
+ if let Schema::Null = schema {
+ visitor.visit_none()
+ } else {
+ visitor.visit_some(self.with_different_schema(schema)?)
+ }
+ } else {
+ Err(self.error("option", "Expected Schema::Union([Schema::Null, _])"))
+ }
+ }
+
+ fn deserialize_unit<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Null => visitor.visit_unit(),
+ Schema::Union(union) => self.with_union(union)?.deserialize_unit(visitor),
+ _ => Err(self.error("unit", "Expected Schema::Null")),
+ }
+ }
+
+ fn deserialize_unit_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ visitor.visit_unit()
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_unit_struct(name, visitor),
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 0)"),
+ )),
+ }
+ }
+
+ fn deserialize_newtype_struct<V>(
+ self,
+ name: &'static str,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == 1 && record.name.name() == name => {
+ visitor.visit_newtype_struct(self.with_different_schema(&record.fields[0].schema)?)
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_newtype_struct(name, visitor),
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == 1)"),
+ )),
+ }
+ }
+
+ fn deserialize_seq<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Array(array) => {
+ visitor.visit_seq(ArrayDeserializer::new(self.reader, array, self.config)?)
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_seq(visitor),
+ _ => Err(self.error("seq", "Expected Schema::Array")),
+ }
+ }
+
+ fn deserialize_tuple<V>(self, len: usize, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // `len == 0` is not possible for derived Deserialize implementations but users might use it.
+ // The derived Deserialize implementations use `deserialize_unit` instead
+ Schema::Null if len == 0 => visitor.visit_unit(),
+ schema if len == 1 => {
+ visitor.visit_seq(OneTupleDeserializer::new(self.reader, schema, self.config)?)
+ }
+ Schema::Record(record) if record.fields.len() == len => {
+ visitor.visit_seq(ManyTupleDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_tuple(len, visitor),
+ _ if len == 0 => Err(self.error("tuple", "Expected Schema::Null for unit tuple")),
+ _ => Err(self.error(
+ "tuple",
+ format!("Expected Schema::Record(fields.len() == {len}) for {len}-tuple"),
+ )),
+ }
+ }
+
+ fn deserialize_tuple_struct<V>(
+ self,
+ name: &'static str,
+ len: usize,
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Record(record) if record.name.name() == name && record.fields.len() == len => {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_tuple_struct(name, len, visitor),
+ _ => Err(self.error(
+ "tuple struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == {len})"),
+ )),
+ }
+ }
+
+ fn deserialize_map<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Map(map) => {
+ visitor.visit_map(MapDeserializer::new(self.reader, map, self.config)?)
+ }
+ Schema::Record(record) => {
+ // Needed for flattened structs which are (de)serialized as maps
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self.with_union(union)?.deserialize_map(visitor),
+ _ => Err(self.error("map", "Expected Schema::Map")),
+ }
+ }
+
+ fn deserialize_struct<V>(
+ self,
+ name: &'static str,
+ fields: &'static [&'static str],
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ // Checking that the amount of fields match does not work because of `skip_deserializing`
+ Schema::Record(record)
+ if record.name.name() == name || name.as_ptr() == DESERIALIZE_ANY.as_ptr() =>
+ {
+ visitor.visit_map(RecordDeserializer::new(self.reader, record, self.config))
+ }
+ Schema::Union(union) => self
+ .with_union(union)?
+ .deserialize_struct(name, fields, visitor),
+ _ => Err(self.error("struct", format!("Expected Schema::Record(name: {name})"))),
+ }
+ }
+
+ fn deserialize_enum<V>(
+ self,
+ _name: &'static str,
+ variants: &'static [&'static str],
+ visitor: V,
+ ) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::Enum(schema) if schema.symbols.len() == variants.len() => {
+ visitor.visit_enum(PlainEnumDeserializer::new(self.reader, schema))
+ }
+ Schema::Union(union) if union.variants().len() == variants.len() => {
+ visitor.visit_enum(UnionEnumDeserializer::new(self.reader, union, self.config))
+ }
+ _ => Err(self.error(
+ "enum",
+ format!(
+ "Expected Schema::Enum(symbols.len() == {}) | Schema::Union(variants.len() == {}",
+ variants.len(), variants.len()
+ ),
+ )),
+ }
+ }
+
+ fn deserialize_identifier<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ match self.schema {
+ Schema::String => self.deserialize_string(visitor),
+ _ => Err(self.error("identifier", "Expected Schema::String")),
+ }
+ }
+
+ fn deserialize_ignored_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+ where
+ V: Visitor<'de>,
+ {
+ // TODO: Add `Seek` bound and skip over ignored data
+ self.deserialize_any(visitor)
+ }
+
+ fn is_human_readable(&self) -> bool {
+ self.config.human_readable
+ }
+}
diff --git a/avro/src/serde/deser_schema/record.rs b/avro/src/serde/deser_schema/record.rs
new file mode 100644
index 0000000..2414037
--- /dev/null
+++ b/avro/src/serde/deser_schema/record.rs
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::schema::RecordSchema;
+use crate::serde::deser_schema::identifier::IdentifierDeserializer;
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
+use crate::{Error, Schema};
+use serde::de::{DeserializeSeed, MapAccess};
+use std::borrow::Borrow;
+use std::io::Read;
+
+pub struct RecordDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> RecordDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> MapAccess<'de> for RecordDeserializer<'s, 'r, R, S> {
+ type Error = Error;
+
+ fn next_key_seed<K>(&mut self, seed: K) -> Result<Option<K::Value>, Self::Error>
+ where
+ K: DeserializeSeed<'de>,
+ {
+ if self.current_field >= self.schema.fields.len() {
+ // Finished reading this record
+ Ok(None)
+ } else {
+ seed.deserialize(IdentifierDeserializer::string(
+ &self.schema.fields[self.current_field].name,
+ ))
+ .map(Some)
+ }
+ }
+
+ fn next_value_seed<V>(&mut self, seed: V) -> Result<V::Value, Self::Error>
+ where
+ V: DeserializeSeed<'de>,
+ {
+ let schema = &self.schema.fields[self.current_field].schema;
+ let value = if let Schema::Record(record) = schema
+ && record.fields.len() == 1
+ && record
+ .attributes
+ .get("org.apache.avro.rust.union_of_records")
+ == Some(&serde_json::Value::Bool(true))
+ {
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ &record.fields[0].schema,
+ self.config,
+ )?)?
+ } else {
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ schema,
+ self.config,
+ )?)?
+ };
+ self.current_field += 1;
+ Ok(value)
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ Some(self.schema.fields.len() - self.current_field)
+ }
+}
diff --git a/avro/src/serde/deser_schema/tuple.rs b/avro/src/serde/deser_schema/tuple.rs
new file mode 100644
index 0000000..5b68746
--- /dev/null
+++ b/avro/src/serde/deser_schema/tuple.rs
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::schema::RecordSchema;
+use crate::serde::deser_schema::{Config, SchemaAwareDeserializer};
+use crate::{Error, Schema};
+use serde::de::{DeserializeSeed, SeqAccess};
+use std::borrow::Borrow;
+use std::io::Read;
+
+pub struct OneTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ field_read: bool,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> OneTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(
+ reader: &'r mut R,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema {
+ config.get_schema(name)?
+ } else {
+ schema
+ };
+ Ok(Self {
+ reader,
+ schema,
+ config,
+ field_read: false,
+ })
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for OneTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.field_read {
+ Ok(None)
+ } else {
+ self.field_read = true;
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ self.schema,
+ self.config,
+ )?)
+ .map(Some)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ Some(1 - usize::from(self.field_read))
+ }
+}
+
+pub struct ManyTupleDeserializer<'s, 'r, R: Read, S: Borrow<Schema>> {
+ reader: &'r mut R,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ current_field: usize,
+}
+
+impl<'s, 'r, R: Read, S: Borrow<Schema>> ManyTupleDeserializer<'s, 'r, R, S> {
+ pub fn new(reader: &'r mut R, schema: &'s RecordSchema, config: Config<'s, S>) -> Self {
+ Self {
+ reader,
+ schema,
+ config,
+ current_field: 0,
+ }
+ }
+}
+
+impl<'de, 's, 'r, R: Read, S: Borrow<Schema>> SeqAccess<'de>
+ for ManyTupleDeserializer<'s, 'r, R, S>
+{
+ type Error = Error;
+
+ fn next_element_seed<T>(&mut self, seed: T) -> Result<Option<T::Value>, Self::Error>
+ where
+ T: DeserializeSeed<'de>,
+ {
+ if self.current_field < self.schema.fields.len() {
+ let schema = &self.schema.fields[self.current_field].schema;
+ self.current_field += 1;
+ seed.deserialize(SchemaAwareDeserializer::new(
+ self.reader,
+ schema,
+ self.config,
+ )?)
+ .map(Some)
+ } else {
+ Ok(None)
+ }
+ }
+
+ fn size_hint(&self) -> Option<usize> {
+ todo!()
+ }
+}
diff --git a/avro/src/serde/mod.rs b/avro/src/serde/mod.rs
index b3bfd2a..256963d 100644
--- a/avro/src/serde/mod.rs
+++ b/avro/src/serde/mod.rs
@@ -109,6 +109,7 @@
mod de;
mod derive;
+pub(crate) mod deser_schema;
mod ser;
pub(crate) mod ser_schema;
mod util;
(avro-rs) 04/05: feat: Rework `SchemaAwareSerializer` to be more strict and follow the documented model (no tests)
Posted by kr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kriskras99 pushed a commit to branch feat/full_enum_support
in repository https://gitbox.apache.org/repos/asf/avro-rs.git
commit 4da2ebf79257820c5321f37221abc62133cb0938
Author: Kriskras99 <gi...@kriskras99.nl>
AuthorDate: Sun Mar 15 23:08:19 2026 +0100
feat: Rework `SchemaAwareSerializer` to be more strict and follow the documented model (no tests)
---
avro/src/documentation/serde_data_model_to_avro.rs | 18 +-
avro/src/error.rs | 7 +
avro/src/schema/union.rs | 101 +-
avro/src/serde/ser_schema/block.rs | 356 +++
avro/src/serde/ser_schema/mod.rs | 3072 +++++++-------------
avro/src/serde/ser_schema/record/field_default.rs | 199 ++
avro/src/serde/ser_schema/record/mod.rs | 191 ++
avro/src/serde/ser_schema/tuple.rs | 278 ++
avro/src/serde/ser_schema/union.rs | 559 ++++
avro/src/writer/datum.rs | 37 +-
avro/src/writer/mod.rs | 34 +-
avro/src/writer/single_object.rs | 46 +-
12 files changed, 2815 insertions(+), 2083 deletions(-)
diff --git a/avro/src/documentation/serde_data_model_to_avro.rs b/avro/src/documentation/serde_data_model_to_avro.rs
index b623577..e590870 100644
--- a/avro/src/documentation/serde_data_model_to_avro.rs
+++ b/avro/src/documentation/serde_data_model_to_avro.rs
@@ -48,17 +48,17 @@
//! - **seq** => [`Schema::Array`]
//! - **tuple**
//! - For tuples with one element, the schema will be the schema the only element
-//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements.
+//! - For tuples with more than one element, the schema will be a [`Schema::Record`] with as many fields as there are elements and in the same order.
//! The schema must have the attribute `org.apache.avro.rust.tuple` with the value set to `true`.
//! - **Note:** Serde (de)serializes arrays (`[T; N]`) as tuples. To (de)serialize an array as a
//! [`Schema::Array`] use [`apache_avro::serde::array`].
-//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements
+//! - **tuple_struct** => [`Schema::Record`] with the unqualified name equal to the name of the struct and as many fields as there are elements and in the same order
//! - **Note:** Tuple structs with 0 or 1 elements will also be (de)serialized as a [`Schema::Record`]. This
//! is different from normal tuples`.
//! - **tuple_variant** => See [Enums](##Enums)
//! - **map** => [`Schema::Map`]
//! - **Note:** the key type of the map will be (de)serialized as a [`Schema::String`]
-//! - **struct** => [`Schema::Record`]
+//! - **struct** => [`Schema::Record`] where the field order must match between the struct and the schema
//! - **struct_variant** => See [Enums](##Enums)
//!
//! ## Enums
@@ -71,13 +71,13 @@
//! - **unit_variant** => [`Schema::Record`] named as the variant and with no fields.
//! - **newtype_variant** => [`Schema::Record`] named as the variant and with one field.
//! The schema must have the attribute `org.apache.avro.rust.union_of_records` with the value set to `true`.
-//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
-//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element and in the same order.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant and in the same order.
//! - As a [`Schema::Union`] without the wrapper [`Schema::Record`], all schemas must be unique:
//! - **unit_variant** => [`Schema::Null`].
//! - **newtype_variant** => The schema of the inner type.
-//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
-//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element and in the same order.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant and in the same order.
//!
//! ### Internally tagged
//! This enum representation is used by Serde if the attribute `#[serde(tag = "...")]` is used.
@@ -95,8 +95,8 @@
//! to a [`Schema::Union`] with the following schemas:
//! - **unit_variant** => [`Schema::Null`].
//! - **newtype_variant** => The schema of the inner type.
-//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element.
-//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant.
+//! - **tuple_variant** => [`Schema::Record`] named as the variant and with as many fields as there are element and in the same order.
+//! - **struct_variant** => [`Schema::Record`] named as the variant and with a field for every field of the struct variant and in the same order.
//!
//! [`AvroSchema`]: crate::AvroSchema
//! [`Schema::Array`]: crate::schema::Schema::Array
diff --git a/avro/src/error.rs b/avro/src/error.rs
index 123bf3c..85c7fd8 100644
--- a/avro/src/error.rs
+++ b/avro/src/error.rs
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+use crate::schema::RecordSchema;
use crate::{
schema::{Name, Schema, SchemaKind, UnionSchema},
types::{Value, ValueKind},
@@ -565,6 +566,12 @@ pub enum Details {
schema: Schema,
},
+ #[error("{position} is not a valid index for fields in {schema:?}")]
+ SerializeRecordUnknownFieldIndex {
+ position: usize,
+ schema: RecordSchema,
+ },
+
#[error("Failed to serialize field '{field_name}' for record {record_schema:?}: {error}")]
SerializeRecordFieldWithSchema {
field_name: String,
diff --git a/avro/src/schema/union.rs b/avro/src/schema/union.rs
index c8bef75..36b7a93 100644
--- a/avro/src/schema/union.rs
+++ b/avro/src/schema/union.rs
@@ -17,7 +17,8 @@
use crate::error::Details;
use crate::schema::{
- DecimalSchema, InnerDecimalSchema, Name, NamespaceRef, Schema, SchemaKind, UuidSchema,
+ DecimalSchema, FixedSchema, InnerDecimalSchema, Name, NamespaceRef, RecordSchema, Schema,
+ SchemaKind, UuidSchema,
};
use crate::types;
use crate::{AvroResult, Error};
@@ -85,6 +86,104 @@ impl UnionSchema {
})
}
+ /// Get the index of the provided [`SchemaKind`].
+ ///
+ /// The schema must not be a logical type, as only the base type are saved in the lookup index.
+ pub(crate) fn index_of_schema_kind(&self, kind: SchemaKind) -> Option<usize> {
+ self.variant_index.get(&kind).copied()
+ }
+
+ /// Get the index and schema for the provided name.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_named_schema<'s>(
+ &'s self,
+ name: &str,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s Schema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ if let Some(schema_name) = schema.name()
+ && schema_name.name() == name
+ {
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ return Ok(Some((index, schema)));
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Fixed`] with the given size.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_fixed_of_size_n<'s>(
+ &'s self,
+ size: usize,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s FixedSchema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ match schema {
+ Schema::Fixed(fixed)
+ | Schema::Uuid(UuidSchema::Fixed(fixed))
+ | Schema::Decimal(DecimalSchema {
+ inner: InnerDecimalSchema::Fixed(fixed),
+ ..
+ })
+ | Schema::Duration(fixed)
+ if fixed.size == size =>
+ {
+ return Ok(Some((index, fixed)));
+ }
+ _ => {}
+ }
+ }
+ Ok(None)
+ }
+
+ /// Find a [`Schema::Record`] with `n` fields.
+ ///
+ /// Will use `names` to resolve references.
+ pub(crate) fn find_record_with_n_fields<'s>(
+ &'s self,
+ n_fields: usize,
+ names: &'s HashMap<Name, impl Borrow<Schema>>,
+ ) -> Result<Option<(usize, &'s RecordSchema)>, Error> {
+ for index in self.named_index.iter().copied() {
+ let schema = &self.schemas[index];
+ let schema = if let Schema::Ref { name } = schema {
+ names
+ .get(name)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()))?
+ .borrow()
+ } else {
+ schema
+ };
+ match schema {
+ Schema::Record(record) if record.fields.len() == n_fields => {
+ return Ok(Some((index, record)));
+ }
+ _ => {}
+ }
+ }
+ Ok(None)
+ }
+
/// Returns true if any of the variants of this `UnionSchema` is `Null`.
pub fn is_nullable(&self) -> bool {
self.variant_index.contains_key(&SchemaKind::Null)
diff --git a/avro/src/serde/ser_schema/block.rs b/avro/src/serde/ser_schema/block.rs
new file mode 100644
index 0000000..e15b486
--- /dev/null
+++ b/avro/src/serde/ser_schema/block.rs
@@ -0,0 +1,356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{Config, SchemaAwareSerializer};
+use crate::encode::encode_int;
+use crate::error::Details;
+use crate::schema::{ArraySchema, MapSchema};
+use crate::util::zig_i32;
+use crate::{Error, Schema};
+use serde::Serialize;
+use serde::ser::{SerializeMap, SerializeSeq};
+use std::borrow::Borrow;
+use std::io::Write;
+
+#[expect(
+ private_interfaces,
+ reason = "Direct and Buffered should not be used directly"
+)]
+pub enum BlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Direct(DirectBlockSerializer<'s, 'w, W, S>),
+ Buffered(BufferedBlockSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BlockSerializer<'s, 'w, W, S> {
+ pub fn array(
+ writer: &'w mut W,
+ schema: &'s ArraySchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.items.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.items
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema.types.as_ref() {
+ config.get_schema(name)?
+ } else {
+ &schema.types
+ };
+
+ Self::new(writer, schema, config, len, bytes_written)
+ }
+
+ fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ let bytes_written = bytes_written.unwrap_or(0);
+ if let Some(len) = len
+ && config.target_block_size.is_none()
+ {
+ Ok(Self::Direct(DirectBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ } else {
+ let target_block_size = config.target_block_size.unwrap_or(1024);
+ Ok(Self::Buffered(BufferedBlockSerializer::new(
+ writer,
+ schema,
+ config,
+ target_block_size,
+ bytes_written,
+ )))
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_element(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_key(key),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ BlockSerializer::Direct(direct) => direct.serialize_value(value),
+ BlockSerializer::Buffered(buffered) => buffered.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ BlockSerializer::Direct(direct) => direct.end(),
+ BlockSerializer::Buffered(buffered) => buffered.end(),
+ }
+ }
+}
+
+struct DirectBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> DirectBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ len: usize,
+ mut bytes_written: usize,
+ ) -> Result<Self, Error> {
+ if len != 0 {
+ // .end() always writes the zero block, so we only write the size for arrays
+ // that have at least one element
+ bytes_written += zig_i32(len as i32, &mut *writer)?;
+ }
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ bytes_written,
+ })
+ }
+
+ fn end(self) -> Result<usize, Error> {
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for DirectBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written += key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+struct BufferedBlockSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ buffer: Vec<u8>,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+ items_in_buffer: usize,
+ target_block_size: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> BufferedBlockSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ target_block_size: usize,
+ bytes_written: usize,
+ ) -> Self {
+ Self {
+ writer,
+ buffer: Vec::with_capacity(target_block_size),
+ schema,
+ config,
+ bytes_written,
+ items_in_buffer: 0,
+ target_block_size,
+ }
+ }
+
+ /// Write a block including the items and size header.
+ fn write_block(&mut self) -> Result<(), Error> {
+ // Write the header, the negative item count indicates that the next value is the size of the
+ // block in bytes
+ self.bytes_written += encode_int(0 - (self.items_in_buffer as i32), &mut *self.writer)?;
+ self.bytes_written += encode_int(self.buffer.len() as i32, &mut *self.writer)?;
+
+ // Write the actual data
+ self.writer
+ .write_all(&self.buffer)
+ .map_err(Details::WriteBytes)?;
+ self.bytes_written += self.buffer.len();
+
+ // Reset the buffer
+ self.items_in_buffer = 0;
+ self.buffer.clear();
+
+ Ok(())
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any items remaining in the buffer
+ if self.items_in_buffer > 0 {
+ self.write_block()?;
+ }
+ debug_assert_eq!(self.buffer.len(), 0, "Buffer must be empty at this point");
+
+ // Write the zero directly instead of through zig_i32 which does a lot of extra work
+ self.writer.write_all(&[0]).map_err(Details::WriteBytes)?;
+
+ Ok(self.bytes_written + 1)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeSeq for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ self.schema,
+ self.config,
+ )?)?;
+ self.items_in_buffer += 1;
+
+ if self.buffer.len() >= self.target_block_size {
+ self.write_block()?;
+ }
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for BufferedBlockSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ key.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &Schema::String,
+ self.config,
+ )?)?;
+ Ok(())
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.serialize_element(value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
diff --git a/avro/src/serde/ser_schema/mod.rs b/avro/src/serde/ser_schema/mod.rs
index fa9382f..829cec4 100644
--- a/avro/src/serde/ser_schema/mod.rs
+++ b/avro/src/serde/ser_schema/mod.rs
@@ -17,1929 +17,398 @@
//! Logic for serde-compatible schema-aware serialization which writes directly to a writer.
-use crate::schema::{DecimalSchema, InnerDecimalSchema, UuidSchema};
-use crate::{
- bigdecimal::big_decimal_as_bytes,
- encode::{encode_int, encode_long},
- error::{Details, Error},
- schema::{Name, NamesRef, Namespace, RecordField, RecordSchema, Schema},
- serde::util::StringSerializer,
+mod block;
+mod record;
+mod tuple;
+mod union;
+
+use crate::error::Details;
+use crate::schema::{
+ DecimalSchema, InnerDecimalSchema, MapSchema, Name, RecordSchema, SchemaKind, UnionSchema,
+ UuidSchema,
};
-use bigdecimal::BigDecimal;
-use serde::{Serialize, ser};
-use std::{cmp::Ordering, collections::HashMap, io::Write, str::FromStr};
-
-const COLLECTION_SERIALIZER_ITEM_LIMIT: usize = 1024;
-const COLLECTION_SERIALIZER_DEFAULT_INIT_ITEM_CAPACITY: usize = 32;
-const SINGLE_VALUE_INIT_BUFFER_SIZE: usize = 128;
-
-/// The sequence serializer for [`SchemaAwareWriteSerializer`].
-///
-/// This may break large arrays up into multiple blocks to avoid having
-/// to obtain the length of the entire array before being able to write any data to the underlying
-/// writer (see the [Data Serialization and Deserialization] for more info).
-/// [Data Serialization and Deserialization](https://avro.apache.org/docs/1.12.0/specification/#data-serialization-and-deserialization)
-pub struct SchemaAwareWriteSerializeSeq<'a, 's, W: Write> {
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- item_schema: &'s Schema,
- item_buffer_size: usize,
- item_buffers: Vec<Vec<u8>>,
- bytes_written: usize,
-}
-
-impl<'a, 's, W: Write> SchemaAwareWriteSerializeSeq<'a, 's, W> {
- fn new(
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- item_schema: &'s Schema,
- len: Option<usize>,
- ) -> SchemaAwareWriteSerializeSeq<'a, 's, W> {
- SchemaAwareWriteSerializeSeq {
- ser,
- item_schema,
- item_buffer_size: SINGLE_VALUE_INIT_BUFFER_SIZE,
- item_buffers: Vec::with_capacity(
- len.unwrap_or(COLLECTION_SERIALIZER_DEFAULT_INIT_ITEM_CAPACITY),
- ),
- bytes_written: 0,
- }
- }
-
- fn write_buffered_items(&mut self) -> Result<(), Error> {
- if !self.item_buffers.is_empty() {
- self.bytes_written +=
- encode_long(self.item_buffers.len() as i64, &mut self.ser.writer)?;
- for item in self.item_buffers.drain(..) {
- self.bytes_written += self
- .ser
- .writer
- .write(item.as_slice())
- .map_err(Details::WriteBytes)?;
- }
- }
-
- Ok(())
- }
-
- fn serialize_element<T: ser::Serialize>(&mut self, value: &T) -> Result<(), Error> {
- let mut item_buffer: Vec<u8> = Vec::with_capacity(self.item_buffer_size);
- let mut item_ser = SchemaAwareWriteSerializer::new(
- &mut item_buffer,
- self.item_schema,
- self.ser.names,
- self.ser.enclosing_namespace.clone(),
- );
- value.serialize(&mut item_ser)?;
-
- self.item_buffer_size = std::cmp::max(self.item_buffer_size, item_buffer.len() + 16);
-
- self.item_buffers.push(item_buffer);
-
- if self.item_buffers.len() > COLLECTION_SERIALIZER_ITEM_LIMIT {
- self.write_buffered_items()?;
- }
-
- Ok(())
- }
-
- fn end(mut self) -> Result<usize, Error> {
- self.write_buffered_items()?;
- self.bytes_written += self.ser.writer.write(&[0u8]).map_err(Details::WriteBytes)?;
-
- Ok(self.bytes_written)
- }
-}
-
-impl<W: Write> ser::SerializeSeq for SchemaAwareWriteSerializeSeq<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- self.serialize_element(&value)
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- self.end()
- }
-}
-
-impl<W: Write> ser::SerializeTuple for SchemaAwareWriteSerializeSeq<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- ser::SerializeSeq::serialize_element(self, value)
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- ser::SerializeSeq::end(self)
- }
-}
-
-/// The map serializer for [`SchemaAwareWriteSerializer`].
-///
-/// This may break large maps up into multiple blocks to avoid having to obtain the size of the entire
-/// map before being able to write any data to the underlying writer
-/// (see [Data Serialization and Deserialization] for more info)
-/// [Data Serialization and Deserialization](https://avro.apache.org/docs/1.12.0/specification/#data-serialization-and-deserialization)
-pub struct SchemaAwareWriteSerializeMap<'a, 's, W: Write> {
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- item_schema: &'s Schema,
- item_buffer_size: usize,
- item_buffers: Vec<Vec<u8>>,
- bytes_written: usize,
-}
-
-impl<'a, 's, W: Write> SchemaAwareWriteSerializeMap<'a, 's, W> {
- fn new(
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- item_schema: &'s Schema,
- len: Option<usize>,
- ) -> SchemaAwareWriteSerializeMap<'a, 's, W> {
- SchemaAwareWriteSerializeMap {
- ser,
- item_schema,
- item_buffer_size: SINGLE_VALUE_INIT_BUFFER_SIZE,
- item_buffers: Vec::with_capacity(
- len.unwrap_or(COLLECTION_SERIALIZER_DEFAULT_INIT_ITEM_CAPACITY),
- ),
- bytes_written: 0,
- }
- }
-
- fn write_buffered_items(&mut self) -> Result<(), Error> {
- if !self.item_buffers.is_empty() {
- self.bytes_written +=
- encode_long(self.item_buffers.len() as i64, &mut self.ser.writer)?;
- for item in self.item_buffers.drain(..) {
- self.bytes_written += self
- .ser
- .writer
- .write(item.as_slice())
- .map_err(Details::WriteBytes)?;
- }
- }
-
- Ok(())
- }
-}
-
-impl<W: Write> ser::SerializeMap for SchemaAwareWriteSerializeMap<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- let mut element_buffer: Vec<u8> = Vec::with_capacity(self.item_buffer_size);
- let string_schema = Schema::String;
- let mut key_ser = SchemaAwareWriteSerializer::new(
- &mut element_buffer,
- &string_schema,
- self.ser.names,
- self.ser.enclosing_namespace.clone(),
- );
- key.serialize(&mut key_ser)?;
-
- self.item_buffers.push(element_buffer);
-
- Ok(())
- }
-
- fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- let last_index = self.item_buffers.len() - 1;
- let element_buffer = &mut self.item_buffers[last_index];
- let mut val_ser = SchemaAwareWriteSerializer::new(
- element_buffer,
- self.item_schema,
- self.ser.names,
- self.ser.enclosing_namespace.clone(),
- );
- value.serialize(&mut val_ser)?;
-
- self.item_buffer_size = std::cmp::max(self.item_buffer_size, element_buffer.len() + 16);
-
- if self.item_buffers.len() > COLLECTION_SERIALIZER_ITEM_LIMIT {
- self.write_buffered_items()?;
- }
-
- Ok(())
- }
-
- fn end(mut self) -> Result<Self::Ok, Self::Error> {
- self.write_buffered_items()?;
- self.bytes_written += self.ser.writer.write(&[0u8]).map_err(Details::WriteBytes)?;
-
- Ok(self.bytes_written)
- }
-}
-
-/// The struct serializer for [`SchemaAwareWriteSerializer`], which can serialize Avro records.
-///
-/// This can accept fields out of order, but doing so incurs a performance penalty, since it requires
-/// buffering serialized values in order to write them to the stream in order.
-pub struct SchemaAwareWriteSerializeStruct<'a, 's, W: Write> {
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- record_schema: &'s RecordSchema,
- /// Fields we received in the wrong order
- field_cache: HashMap<usize, Vec<u8>>,
- /// The current field name when serializing from a map (for `flatten` support).
- map_field_name: Option<String>,
- field_position: usize,
- bytes_written: usize,
-}
-
-impl<'a, 's, W: Write> SchemaAwareWriteSerializeStruct<'a, 's, W> {
- fn new(
- ser: &'a mut SchemaAwareWriteSerializer<'s, W>,
- record_schema: &'s RecordSchema,
- ) -> SchemaAwareWriteSerializeStruct<'a, 's, W> {
- SchemaAwareWriteSerializeStruct {
- ser,
- record_schema,
- field_cache: HashMap::new(),
- map_field_name: None,
- field_position: 0,
- bytes_written: 0,
- }
- }
-
- fn serialize_next_field<T>(
- &mut self,
- field: &RecordField,
- position: usize,
- value: &T,
- ) -> Result<(), Error>
- where
- T: ?Sized + ser::Serialize,
- {
- match self.field_position.cmp(&position) {
- Ordering::Equal => {
- // If we receive fields in order, write them directly to the main writer
- let mut value_ser = SchemaAwareWriteSerializer::new(
- &mut *self.ser.writer,
- &field.schema,
- self.ser.names,
- self.ser.enclosing_namespace.clone(),
- );
- self.bytes_written += value.serialize(&mut value_ser)?;
-
- self.field_position += 1;
- while let Some(bytes) = self.field_cache.remove(&self.field_position) {
- self.ser
- .writer
- .write_all(&bytes)
- .map_err(Details::WriteBytes)?;
- self.bytes_written += bytes.len();
- self.field_position += 1;
- }
- Ok(())
- }
- Ordering::Less => {
- // Current field position is smaller than this field position,
- // so we're still missing at least one field, save this field temporarily
- let mut bytes = Vec::new();
- let mut value_ser = SchemaAwareWriteSerializer::new(
- &mut bytes,
- &field.schema,
- self.ser.names,
- self.ser.enclosing_namespace.clone(),
- );
- value.serialize(&mut value_ser)?;
- if self.field_cache.insert(position, bytes).is_some() {
- Err(Details::FieldNameDuplicate(field.name.clone()).into())
- } else {
- Ok(())
- }
- }
- Ordering::Greater => {
- // Current field position is greater than this field position,
- // so we've already had this field
- Err(Details::FieldNameDuplicate(field.name.clone()).into())
- }
- }
- }
-
- fn end(mut self) -> Result<usize, Error> {
- // Write any fields that are `serde(skip)` or `serde(skip_serializing)`
- while self.field_position != self.record_schema.fields.len() {
- let field_info = &self.record_schema.fields[self.field_position];
- if let Some(bytes) = self.field_cache.remove(&self.field_position) {
- self.ser
- .writer
- .write_all(&bytes)
- .map_err(Details::WriteBytes)?;
- self.bytes_written += bytes.len();
- self.field_position += 1;
- } else if let Some(default) = &field_info.default {
- self.serialize_next_field(field_info, self.field_position, default)
- .map_err(|e| Details::SerializeRecordFieldWithSchema {
- field_name: field_info.name.clone(),
- record_schema: Schema::Record(self.record_schema.clone()),
- error: Box::new(e),
- })?;
- } else {
- return Err(Details::MissingDefaultForSkippedField {
- field_name: field_info.name.clone(),
- schema: Schema::Record(self.record_schema.clone()),
- }
- .into());
- }
- }
-
- debug_assert!(
- self.field_cache.is_empty(),
- "There should be no more unwritten fields at this point: {:?}",
- self.field_cache
- );
- debug_assert!(
- self.map_field_name.is_none(),
- "There should be no field name at this point: field {:?}",
- self.map_field_name
- );
- Ok(self.bytes_written)
- }
-}
-
-impl<W: Write> ser::SerializeStruct for SchemaAwareWriteSerializeStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- if let Some(position) = self.record_schema.lookup.get(key).copied() {
- let field = self
- .record_schema
- .fields
- .get(position)
- .ok_or_else(|| Details::FieldName(String::from(key)))?;
- self.serialize_next_field(field, position, value)
- .map_err(|e| {
- Details::SerializeRecordFieldWithSchema {
- field_name: key.to_string(),
- record_schema: Schema::Record(self.record_schema.clone()),
- error: Box::new(e),
- }
- .into()
- })
- } else {
- Err(Details::FieldName(String::from(key)).into())
- }
- }
-
- fn skip_field(&mut self, key: &'static str) -> Result<(), Self::Error> {
- if let Some(position) = self.record_schema.lookup.get(key).copied() {
- let field = &self.record_schema.fields[position];
- if let Some(default) = &field.default {
- self.serialize_next_field(field, position, default)
- .map_err(|e| {
- Details::SerializeRecordFieldWithSchema {
- field_name: key.to_string(),
- record_schema: Schema::Record(self.record_schema.clone()),
- error: Box::new(e),
- }
- .into()
- })
- } else {
- Err(Details::MissingDefaultForSkippedField {
- field_name: key.to_string(),
- schema: Schema::Record(self.record_schema.clone()),
- }
- .into())
- }
- } else {
- Err(Details::GetField(key.to_string()).into())
- }
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- self.end()
- }
-}
-
-/// This implementation is used to support `#[serde(flatten)]` as that uses [`SerializeMap`] instead of [`SerializeStruct`].
-///
-/// [`SerializeMap`]: ser::SerializeMap
-/// [`SerializeStruct`]: ser::SerializeStruct
-impl<W: Write> ser::SerializeMap for SchemaAwareWriteSerializeStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + Serialize,
- {
- let name = key.serialize(StringSerializer)?;
- let old = self.map_field_name.replace(name);
- debug_assert!(
- old.is_none(),
- "Expected a value instead of a key: old key: {old:?}, new key: {:?}",
- self.map_field_name
- );
- Ok(())
- }
-
- fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + Serialize,
- {
- let key = self.map_field_name.take().ok_or(Details::MapNoKey)?;
- if let Some(position) = self.record_schema.lookup.get(&key).copied() {
- let field = &self.record_schema.fields[position];
- self.serialize_next_field(field, position, value)
- .map_err(|e| {
- Details::SerializeRecordFieldWithSchema {
- field_name: key.to_string(),
- record_schema: Schema::Record(self.record_schema.clone()),
- error: Box::new(e),
- }
- .into()
- })
- } else {
- Err(Details::FieldName(key).into())
- }
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- self.end()
- }
-}
-
-impl<W: Write> ser::SerializeStructVariant for SchemaAwareWriteSerializeStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- ser::SerializeStruct::serialize_field(self, key, value)
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- ser::SerializeStruct::end(self)
- }
-}
-
-/// Map serializer that switches between Struct or Map.
-///
-/// This exists because when `#[serde(flatten)]` is used, struct fields are serialized as a map.
-pub enum SchemaAwareWriteSerializeMapOrStruct<'a, 's, W: Write> {
- Struct(SchemaAwareWriteSerializeStruct<'a, 's, W>),
- Map(SchemaAwareWriteSerializeMap<'a, 's, W>),
-}
-
-impl<W: Write> ser::SerializeMap for SchemaAwareWriteSerializeMapOrStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + Serialize,
- {
- match self {
- Self::Struct(s) => s.serialize_key(key),
- Self::Map(s) => s.serialize_key(key),
- }
- }
-
- fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + Serialize,
- {
- match self {
- Self::Struct(s) => s.serialize_value(value),
- Self::Map(s) => s.serialize_value(value),
- }
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- match self {
- Self::Struct(s) => s.end(),
- Self::Map(s) => s.end(),
- }
- }
-}
-
-/// The tuple struct serializer for [`SchemaAwareWriteSerializer`].
-///
-/// This can serialize to an Avro array, record, or big-decimal.
-/// When serializing to a record, fields must be provided in the correct order, since no names are provided.
-pub enum SchemaAwareWriteSerializeTupleStruct<'a, 's, W: Write> {
- Record(SchemaAwareWriteSerializeStruct<'a, 's, W>),
- Array(SchemaAwareWriteSerializeSeq<'a, 's, W>),
-}
-
-impl<W: Write> SchemaAwareWriteSerializeTupleStruct<'_, '_, W> {
- fn serialize_field<T>(&mut self, value: &T) -> Result<(), Error>
- where
- T: ?Sized + ser::Serialize,
- {
- use SchemaAwareWriteSerializeTupleStruct::*;
- match self {
- Record(_record_ser) => {
- unimplemented!("Tuple struct serialization to record is not supported!");
- }
- Array(array_ser) => array_ser.serialize_element(&value),
- }
- }
-
- fn end(self) -> Result<usize, Error> {
- use SchemaAwareWriteSerializeTupleStruct::*;
- match self {
- Record(record_ser) => record_ser.end(),
- Array(array_ser) => array_ser.end(),
- }
- }
-}
-
-impl<W: Write> ser::SerializeTupleStruct for SchemaAwareWriteSerializeTupleStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_field<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- self.serialize_field(&value)
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- self.end()
- }
-}
-
-impl<W: Write> ser::SerializeTupleVariant for SchemaAwareWriteSerializeTupleStruct<'_, '_, W> {
- type Ok = usize;
- type Error = Error;
-
- fn serialize_field<T>(&mut self, value: &T) -> Result<(), Self::Error>
- where
- T: ?Sized + ser::Serialize,
- {
- self.serialize_field(&value)
- }
-
- fn end(self) -> Result<Self::Ok, Self::Error> {
- self.end()
- }
-}
-
-/// A [`Serializer`](ser::Serializer) implementation that serializes directly to raw Avro data.
-///
-/// If data does not match with the schema it will return an error.
-///
-/// This does not keep state and can therefore be reused to write to the same writer.
-pub struct SchemaAwareWriteSerializer<'s, W: Write> {
- writer: &'s mut W,
- root_schema: &'s Schema,
- names: &'s NamesRef<'s>,
- enclosing_namespace: Namespace,
-}
-
-impl<'s, W: Write> SchemaAwareWriteSerializer<'s, W> {
- /// Create a new [`SchemaAwareWriteSerializer`].
+use crate::util::{zig_i32, zig_i64};
+use crate::{Error, Schema};
+use serde::ser::SerializeMap;
+use serde::{Serialize, Serializer};
+use serde_json::Value::Bool;
+use std::borrow::Borrow;
+use std::collections::HashMap;
+use std::io::Write;
+
+use block::BlockSerializer;
+use record::RecordSerializer;
+use tuple::{ManyTupleSerializer, TupleSerializer};
+use union::UnionSerializer;
+
+pub struct Config<'s, S: Borrow<Schema>> {
+ /// Any references in the schema will be resolved using this map.
///
- /// `writer` is the [`std::fmt::Write`] stream to be written to.
+ /// This map is not allowed to contain any [`Schema::Ref`], the serializer is allowed to panic
+ /// in that case.
+ pub names: &'s HashMap<Name, S>,
+ /// At what block size to start a new block (for arrays and maps).
///
- /// `schema` is the schema of the value to be written.
+ /// This is a minimum value, the block size will always be larger than this except for the last
+ /// block.
///
- /// `names` is the mapping of schema names to schemas, to be used for type reference lookups
+ /// When set to `None` all values will be written in a single block. This can be faster as no
+ /// intermediate buffer is used, but seeking through written data will be slower.
+ pub target_block_size: Option<usize>,
+ /// Should `Serialize` implementations pick a human-readable format.
///
- /// `enclosing_namespace` is the enclosing namespace to be used for type reference lookups
- pub fn new(
- writer: &'s mut W,
- schema: &'s Schema,
- names: &'s NamesRef<'s>,
- enclosing_namespace: Namespace,
- ) -> SchemaAwareWriteSerializer<'s, W> {
- SchemaAwareWriteSerializer {
- writer,
- root_schema: schema,
- names,
- enclosing_namespace,
- }
- }
-
- fn get_ref_schema(&self, name: &'s Name) -> Result<&'s Schema, Error> {
- let full_name = name.fully_qualified_name(self.enclosing_namespace.as_deref());
-
- let ref_schema = self.names.get(full_name.as_ref()).copied();
-
- ref_schema.ok_or_else(|| Details::SchemaResolutionError(full_name.as_ref().clone()).into())
- }
-
- fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
- let mut bytes_written: usize = 0;
-
- bytes_written += encode_long(bytes.len() as i64, &mut self.writer)?;
- // write_all() will retry when the error is ErrorKind::Interrupted (happens mostly on network storage)
- self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
- bytes_written += bytes.len();
-
- Ok(bytes_written)
- }
-
- fn serialize_bool_with_schema(&mut self, value: bool, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "bool",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Boolean => self
- .writer
- .write(&[u8::from(value)])
- .map_err(|e| Details::WriteBytes(e).into()),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Boolean => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bool_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "No matching Schema::Bool found in {:?}",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: Bool"))),
- }
- }
-
- fn serialize_i32_with_schema(&mut self, value: i32, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "int (i8 | i16 | i32)",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Int | Schema::TimeMillis | Schema::Date => encode_int(value, &mut self.writer),
- Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => encode_long(value as i64, &mut self.writer),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Int
- | Schema::TimeMillis
- | Schema::Date
- | Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_i32_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching int-like schema in {union_schema:?}"
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: Int/Long"))),
- }
- }
-
- fn serialize_i64_with_schema(&mut self, value: i64, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "i64",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Int | Schema::TimeMillis | Schema::Date => {
- let int_value =
- i32::try_from(value).map_err(|cause| create_error(cause.to_string()))?;
- encode_int(int_value, &mut self.writer)
- }
- Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => encode_long(value, &mut self.writer),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Int
- | Schema::TimeMillis
- | Schema::Date
- | Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_i64_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching int/long-like schema in {:?}",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Int/Long"))),
- }
- }
-
- fn serialize_i128_with_schema(&mut self, value: i128, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "i128",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
- self.writer
- .write_all(&value.to_le_bytes())
- .map_err(Details::WriteBytes)?;
- Ok(16)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_i128_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- r#"Cannot find a Fixed(size = 16, name = "i128") schema in {:?}"#,
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: i128"))),
- }
- }
-
- fn serialize_u8_with_schema(&mut self, value: u8, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "u8",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Int | Schema::TimeMillis | Schema::Date => {
- encode_int(value as i32, &mut self.writer)
- }
- Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => encode_long(value as i64, &mut self.writer),
- Schema::Bytes => self.write_bytes(&[value]),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Int
- | Schema::TimeMillis
- | Schema::Date
- | Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos
- | Schema::Bytes => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_u8_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching Int-like, Long-like or Bytes schema in {union_schema:?}"
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Int"))),
- }
- }
-
- fn serialize_u32_with_schema(&mut self, value: u32, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "unsigned int (u16 | u32)",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Int | Schema::TimeMillis | Schema::Date => {
- let int_value =
- i32::try_from(value).map_err(|cause| create_error(cause.to_string()))?;
- encode_int(int_value, &mut self.writer)
- }
- Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => encode_long(value as i64, &mut self.writer),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Int
- | Schema::TimeMillis
- | Schema::Date
- | Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_u32_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching Int-like or Long-like schema in {union_schema:?}"
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Int/Long"))),
- }
- }
-
- fn serialize_u64_with_schema(&mut self, value: u64, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "u64",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Int | Schema::TimeMillis | Schema::Date => {
- let int_value =
- i32::try_from(value).map_err(|cause| create_error(cause.to_string()))?;
- encode_int(int_value, &mut self.writer)
- }
- Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => {
- let long_value =
- i64::try_from(value).map_err(|cause| create_error(cause.to_string()))?;
- encode_long(long_value, &mut self.writer)
- }
- Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
- self.writer
- .write_all(&value.to_le_bytes())
- .map_err(Details::WriteBytes)?;
- Ok(8)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Int
- | Schema::TimeMillis
- | Schema::Date
- | Schema::Long
- | Schema::TimeMicros
- | Schema::TimestampMillis
- | Schema::TimestampMicros
- | Schema::TimestampNanos
- | Schema::LocalTimestampMillis
- | Schema::LocalTimestampMicros
- | Schema::LocalTimestampNanos => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_u64_with_schema(value, variant_schema);
- }
- Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_u64_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- r#"Cannot find a matching Int-like, Long-like or Fixed(size = 8, name "u64") schema in {:?}"#,
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: u64"))),
- }
- }
-
- fn serialize_u128_with_schema(&mut self, value: u128, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "u128",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
- self.writer
- .write_all(&value.to_le_bytes())
- .map_err(Details::WriteBytes)?;
- Ok(16)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_u128_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- r#"Cannot find a Fixed(size = 16, name = "u128") schema in {:?}"#,
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: u128"))),
- }
- }
-
- fn serialize_f32_with_schema(&mut self, value: f32, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "f32",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Float => self
- .writer
- .write(&value.to_le_bytes())
- .map_err(|e| Details::WriteBytes(e).into()),
- Schema::Double => self
- .writer
- .write(&(value as f64).to_le_bytes())
- .map_err(|e| Details::WriteBytes(e).into()),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Float | Schema::Double => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_f32_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a Float schema in {:?}",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Float"))),
- }
- }
-
- fn serialize_f64_with_schema(&mut self, value: f64, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "f64",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Float => self
- .writer
- .write(&(value as f32).to_le_bytes())
- .map_err(|e| Details::WriteBytes(e).into()),
- Schema::Double => self
- .writer
- .write(&value.to_le_bytes())
- .map_err(|e| Details::WriteBytes(e).into()),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Float | Schema::Double => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_f64_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a Double schema in {:?}",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Double"))),
- }
- }
-
- fn serialize_char_with_schema(&mut self, value: char, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "char",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::String | Schema::Bytes => self.write_bytes(String::from(value).as_bytes()),
- Schema::Fixed(fixed) if fixed.size == 4 && fixed.name.name() == "char" => {
- self.writer
- .write_all(&u32::from(value).to_le_bytes())
- .map_err(Details::WriteBytes)?;
- Ok(4)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::String | Schema::Bytes => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_char_with_schema(value, variant_schema);
- }
- Schema::Fixed(fixed) if fixed.size == 4 && fixed.name.name() == "char" => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_char_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- r#"Cannot find a matching String, Bytes or Fixed(size = 4, name = "char") schema in {union_schema:?}"#
- )))
- }
- expected => Err(create_error(format!("Expected {expected}. Got: char"))),
- }
- }
-
- fn serialize_str_with_schema(&mut self, value: &str, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "string",
- value: format!("{value}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::String | Schema::Bytes | Schema::Uuid(UuidSchema::String) => {
- self.write_bytes(value.as_bytes())
- }
- Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_)) => {
- Err(create_error("Expected bytes but got a string. Did you mean to use `Schema::Uuid(UuidSchema::String)` or `utils::serde_set_human_readable(false)`?".to_string()))
- }
- Schema::BigDecimal => {
- // If we get a string for a `BigDecimal` type, expect a display string representation, such as "12.75"
- let decimal_val =
- BigDecimal::from_str(value).map_err(|e| create_error(e.to_string()))?;
- let decimal_bytes = big_decimal_as_bytes(&decimal_val)?;
- self.write_bytes(decimal_bytes.as_slice())
- }
- Schema::Fixed(fixed_schema) => {
- if value.len() == fixed_schema.size {
- self.writer
- .write(value.as_bytes())
- .map_err(|e| Details::WriteBytes(e).into())
- } else {
- Err(create_error(format!(
- "Fixed schema size ({}) does not match the value length ({})",
- fixed_schema.size,
- value.len()
- )))
- }
- }
- Schema::Ref { name } => {
- let ref_schema = self.get_ref_schema(name)?;
- self.serialize_str_with_schema(value, ref_schema)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::String
- | Schema::Bytes
- | Schema::Uuid(UuidSchema::String)
- | Schema::Fixed(_)
- | Schema::Ref { name: _ } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_str_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected one of the union variants {:?}. Got: String",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: String"))),
- }
- }
-
- fn serialize_bytes_with_schema(
- &mut self,
- value: &[u8],
- schema: &Schema,
- ) -> Result<usize, Error> {
- let create_error = |cause: String| {
- use std::fmt::Write;
- let mut v_str = String::with_capacity(value.len());
- for b in value {
- if write!(&mut v_str, "{b:x}").is_err() {
- v_str.push_str("??");
- }
- }
- Error::new(Details::SerializeValueWithSchema {
- value_type: "bytes",
- value: format!("{v_str}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::String | Schema::Bytes | Schema::BigDecimal => self.write_bytes(value),
- Schema::Uuid(UuidSchema::Bytes) => {
- if value.len() == 16 {
- self.write_bytes(value)
- } else {
- Err(create_error(format!(
- "Expected 16 bytes for `Schema::Uuid(Bytes)` but got {} bytes",
- value.len()
- )))
- }
- }
- Schema::Uuid(UuidSchema::String) if value.len() == 16 => {
- Err(create_error("Expected a string, but got 16 bytes. Did you mean to use `Schema::Uuid(UuidSchema::Fixed)` or `utils::serde_set_human_readable(true)`?".to_string()))
- }
- Schema::Fixed(fixed_schema) | Schema::Uuid(UuidSchema::Fixed(fixed_schema)) => {
- if value.len() == fixed_schema.size {
- self.writer
- .write(value)
- .map_err(|e| Details::WriteBytes(e).into())
- } else {
- Err(create_error(format!(
- "Fixed schema size ({}) does not match the value length ({})",
- fixed_schema.size,
- value.len()
- )))
- }
- }
- Schema::Duration(_) => {
- if value.len() == 12 {
- self.writer
- .write(value)
- .map_err(|e| Details::WriteBytes(e).into())
- } else {
- Err(create_error(format!(
- "Duration length must be 12! Got ({})",
- value.len()
- )))
- }
- }
- Schema::Decimal(decimal_schema) => match &decimal_schema.inner {
- InnerDecimalSchema::Bytes => self.write_bytes(value),
- InnerDecimalSchema::Fixed(fixed_schema) => {
- match fixed_schema.size.checked_sub(value.len()) {
- Some(pad) => {
- let pad_val = match value.len() {
- 0 => 0,
- _ => value[0],
- };
- let padding = vec![pad_val; pad];
- let mut bytes_written = self
- .writer
- .write(padding.as_slice())
- .map_err(Details::WriteBytes)?;
- bytes_written +=
- self.writer.write(value).map_err(Details::WriteBytes)?;
- Ok(bytes_written)
- }
- None => Err(Details::CompareFixedSizes {
- size: fixed_schema.size,
- n: value.len(),
- }
- .into()),
- }
- }
- },
- Schema::Ref { name } => {
- let ref_schema = self.get_ref_schema(name)?;
- self.serialize_bytes_with_schema(value, ref_schema)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::String
- | Schema::Bytes
- | Schema::BigDecimal
- | Schema::Decimal(DecimalSchema {
- inner: InnerDecimalSchema::Bytes,
- ..
- })
- | Schema::Ref { name: _ } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bytes_with_schema(value, variant_schema);
- }
- Schema::Uuid(UuidSchema::Bytes) if value.len() == 16 => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bytes_with_schema(value, variant_schema);
- }
- Schema::Fixed(fixed) | Schema::Uuid(UuidSchema::Fixed(fixed))
- if fixed.size == value.len() =>
- {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bytes_with_schema(value, variant_schema);
- }
- Schema::Decimal(DecimalSchema {
- inner: InnerDecimalSchema::Fixed(fixed),
- ..
- }) if fixed.size >= value.len() => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bytes_with_schema(value, variant_schema);
- }
- Schema::Duration(_) if value.len() == 12 => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_bytes_with_schema(value, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching String, Bytes, Uuid, BigDecimal, Fixed, Duration, Decimal or Ref schema in {union_schema:?}"
- )))
- }
- unsupported => Err(create_error(format!(
- "Expected String, Bytes, Uuid, BigDecimal, Fixed, Duration, Decimal, Ref or Union schema. Got: {unsupported}"
- ))),
- }
- }
-
- fn serialize_none_with_schema(&mut self, schema: &Schema) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "none",
- value: format!("None. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Null => Ok(0),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Null => {
- return encode_int(i as i32, &mut *self.writer);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching Null schema in {:?}",
- union_schema.schemas
- )))
- }
- expected => Err(create_error(format!("Expected: {expected}. Got: Null"))),
- }
- }
-
- fn serialize_some_with_schema<T>(&mut self, value: &T, schema: &Schema) -> Result<usize, Error>
- where
- T: ?Sized + ser::Serialize,
- {
- let mut inner_ser = SchemaAwareWriteSerializer::new(
- &mut *self.writer,
- schema,
- self.names,
- self.enclosing_namespace.clone(),
- );
- value.serialize(&mut inner_ser)
- }
-
- fn serialize_unit_struct_with_schema(
- &mut self,
- name: &'static str,
- schema: &Schema,
- ) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "unit struct",
- value: format!("{name}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Record(sch) => match sch.fields.len() {
- 0 => Ok(0),
- too_many => Err(create_error(format!(
- "Too many fields: {too_many}. Expected: 0"
- ))),
- },
- Schema::Null => Ok(0),
- Schema::Ref { name: ref_name } => {
- let ref_schema = self.get_ref_schema(ref_name)?;
- self.serialize_unit_struct_with_schema(name, ref_schema)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Record(record_schema) if record_schema.fields.is_empty() => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_unit_struct_with_schema(name, variant_schema);
- }
- Schema::Null | Schema::Ref { name: _ } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_unit_struct_with_schema(name, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Cannot find a matching Null schema in {union_schema:?}"
- )))
- }
- unsupported => Err(create_error(format!(
- "Expected Null or Union schema. Got: {unsupported}"
- ))),
- }
- }
-
- fn serialize_unit_variant_with_schema(
- &mut self,
- name: &'static str,
- variant_index: u32,
- variant: &'static str,
- schema: &Schema,
- ) -> Result<usize, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "unit variant",
- value: format!("{name}::{variant} (index={variant_index}). Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Enum(enum_schema) => {
- if variant_index as usize >= enum_schema.symbols.len() {
- return Err(create_error(format!(
- "Variant index out of bounds: {}. The Enum schema has '{}' symbols",
- variant_index,
- enum_schema.symbols.len()
- )));
- }
-
- encode_int(variant_index as i32, &mut self.writer)
- }
- Schema::Union(union_schema) => {
- if variant_index as usize >= union_schema.schemas.len() {
- return Err(create_error(format!(
- "Variant index out of bounds: {}. The union schema has '{}' schemas",
- variant_index,
- union_schema.schemas.len()
- )));
- }
-
- encode_int(variant_index as i32, &mut self.writer)?;
- self.serialize_unit_struct_with_schema(
- name,
- &union_schema.schemas[variant_index as usize],
- )
- }
- Schema::Ref { name: ref_name } => {
- let ref_schema = self.get_ref_schema(ref_name)?;
- self.serialize_unit_variant_with_schema(name, variant_index, variant, ref_schema)
- }
- unsupported => Err(create_error(format!(
- "Unsupported schema: {unsupported:?}. Expected: Enum, Union or Ref"
- ))),
- }
- }
-
- fn serialize_newtype_struct_with_schema<T>(
- &mut self,
- _name: &'static str,
- value: &T,
- schema: &Schema,
- ) -> Result<usize, Error>
- where
- T: ?Sized + ser::Serialize,
- {
- let mut inner_ser = SchemaAwareWriteSerializer::new(
- &mut *self.writer,
- schema,
- self.names,
- self.enclosing_namespace.clone(),
- );
- // Treat any newtype struct as a transparent wrapper around the contained type
- value.serialize(&mut inner_ser)
- }
-
- fn serialize_newtype_variant_with_schema<T>(
- &mut self,
- name: &'static str,
- variant_index: u32,
- variant: &'static str,
- value: &T,
- schema: &Schema,
- ) -> Result<usize, Error>
- where
- T: ?Sized + ser::Serialize,
- {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "newtype variant",
- value: format!("{name}::{variant}(?) (index={variant_index}). Cause: {cause}"),
- schema: schema.clone(),
- })
- };
+ /// It is recommended to set this to `false` as it results in compacter output.
+ pub human_readable: bool,
+}
- match schema {
- Schema::Union(union_schema) => {
- let variant_schema = union_schema
- .schemas
- .get(variant_index as usize)
- .ok_or_else(|| {
- create_error(format!(
- "No variant schema at position {variant_index} for {union_schema:?}"
- ))
- })?;
-
- encode_int(variant_index as i32, &mut self.writer)?;
- self.serialize_newtype_struct_with_schema(variant, value, variant_schema)
- }
- _ => Err(create_error(format!(
- "Expected Union schema. Got: {schema}"
- ))),
- }
+impl<'s, S: Borrow<Schema>> Config<'s, S> {
+ /// Get the schema for this name.
+ fn get_schema(&self, name: &Name) -> Result<&'s Schema, Error> {
+ self.names
+ .get(name)
+ .map(Borrow::borrow)
+ .ok_or_else(|| Details::SchemaResolutionError(name.clone()).into())
}
+}
- fn serialize_seq_with_schema<'a>(
- &'a mut self,
- len: Option<usize>,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeSeq<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- let len_str = len
- .map(|l| format!("{l}"))
- .unwrap_or_else(|| String::from("?"));
-
- Error::new(Details::SerializeValueWithSchema {
- value_type: "sequence",
- value: format!("sequence (len={len_str}). Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Array(array_schema) => Ok(SchemaAwareWriteSerializeSeq::new(
- self,
- array_schema.items.as_ref(),
- len,
- )),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Array(_) => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_seq_with_schema(len, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected Array schema in {union_schema:?}"
- )))
- }
- _ => Err(create_error(format!("Expected: {schema}. Got: Array"))),
- }
+// This needs to be implemented manually as the derive puts a bound on `S`
+// which is unnecessary as a reference is always Copy.
+impl<'s, S: Borrow<Schema>> Copy for Config<'s, S> {}
+impl<'s, S: Borrow<Schema>> Clone for Config<'s, S> {
+ fn clone(&self) -> Self {
+ *self
}
+}
- fn serialize_tuple_with_schema<'a>(
- &'a mut self,
- len: usize,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeSeq<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "tuple",
- value: format!("tuple (len={len}). Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Array(array_schema) => Ok(SchemaAwareWriteSerializeSeq::new(
- self,
- array_schema.items.as_ref(),
- Some(len),
- )),
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Array(_) => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_tuple_with_schema(len, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected Array schema in {union_schema:?}"
- )))
- }
- _ => Err(create_error(format!("Expected: {schema}. Got: Array"))),
- }
- }
+pub struct SchemaAwareSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ /// The schema of the data being serialized.
+ ///
+ /// This schema is guaranteed to not be a [`Schema::Ref`].
+ schema: &'s Schema,
+ config: Config<'s, S>,
+}
- fn serialize_tuple_struct_with_schema<'a>(
- &'a mut self,
- name: &'static str,
- len: usize,
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SchemaAwareSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeTupleStruct<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "tuple struct",
- value: format!(
- "{name}({}). Cause: {cause}",
- vec!["?"; len].as_slice().join(",")
- ),
- schema: schema.clone(),
- })
+ config: Config<'s, S>,
+ ) -> Result<Self, Error> {
+ let schema = if let Schema::Ref { name } = schema {
+ config.get_schema(name)?
+ } else {
+ schema
};
+ Ok(Self {
+ writer,
+ schema,
+ config,
+ })
+ }
- match schema {
- Schema::Array(sch) => Ok(SchemaAwareWriteSerializeTupleStruct::Array(
- SchemaAwareWriteSerializeSeq::new(self, &sch.items, Some(len)),
- )),
- Schema::Record(sch) => Ok(SchemaAwareWriteSerializeTupleStruct::Record(
- SchemaAwareWriteSerializeStruct::new(self, sch),
- )),
- Schema::Ref { name: ref_name } => {
- let ref_schema = self.get_ref_schema(ref_name)?;
- self.serialize_tuple_struct_with_schema(name, len, ref_schema)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Record(inner) => {
- if inner.fields.len() == len {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_tuple_struct_with_schema(
- name,
- len,
- variant_schema,
- );
- }
- }
- Schema::Array(_) | Schema::Ref { name: _ } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_tuple_struct_with_schema(
- name,
- len,
- variant_schema,
- );
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected Record, Array or Ref schema in {union_schema:?}"
- )))
- }
- _ => Err(create_error(format!(
- "Expected Record, Array, Ref or Union schema. Got: {schema}"
- ))),
- }
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: self.schema.clone(),
+ })
}
- fn serialize_tuple_variant_with_schema<'a>(
- &'a mut self,
- name: &'static str,
- variant_index: u32,
- variant: &'static str,
- len: usize,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeTupleStruct<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "tuple variant",
- value: format!(
- "{name}::{variant}({}) (index={variant_index}). Cause: {cause}",
- vec!["?"; len].as_slice().join(",")
- ),
- schema: schema.clone(),
- })
+ /// Create a new serializer with the existing writer and config.
+ ///
+ /// This will resolve the schema if it is a reference.
+ fn with_different_schema(mut self, schema: &'s Schema) -> Result<Self, Error> {
+ self.schema = if let Schema::Ref { name } = schema {
+ self.config.get_schema(name)?
+ } else {
+ schema
};
+ Ok(self)
+ }
- match schema {
- Schema::Union(union_schema) => {
- let variant_schema = union_schema
- .schemas
- .get(variant_index as usize)
- .ok_or_else(|| {
- create_error(format!(
- "Cannot find a variant at position {variant_index} in {union_schema:?}"
- ))
- })?;
-
- encode_int(variant_index as i32, &mut self.writer)?;
- self.serialize_tuple_struct_with_schema(variant, len, variant_schema)
- }
- _ => Err(create_error(format!(
- "Expected Union schema. Got: {schema}"
- ))),
+ /// Get the schema at the given index of the union, resolving references.
+ fn get_resolved_union_variant(
+ &self,
+ union: &'s UnionSchema,
+ index: u32,
+ ) -> Result<&'s Schema, Error> {
+ match union.get_variant(index as usize)? {
+ Schema::Ref { name } => self.config.get_schema(name),
+ schema => Ok(schema),
}
}
- fn serialize_map_with_schema<'a>(
- &'a mut self,
- len: Option<usize>,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeMapOrStruct<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- let len_str = len
- .map(|l| format!("{l}"))
- .unwrap_or_else(|| String::from("?"));
-
- Error::new(Details::SerializeValueWithSchema {
- value_type: "map",
- value: format!("map (size={len_str}). Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Map(map_schema) => Ok(SchemaAwareWriteSerializeMapOrStruct::Map(
- SchemaAwareWriteSerializeMap::new(self, map_schema.types.as_ref(), len),
- )),
- Schema::Ref { name: ref_name } => {
- let ref_schema = self.get_ref_schema(ref_name)?;
- self.serialize_map_with_schema(len, ref_schema)
- }
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Map(_) | Schema::Record(_) | Schema::Ref { .. } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_map_with_schema(len, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected a Map schema in {union_schema:?}"
- )))
- }
- Schema::Record(record_schema) => Ok(SchemaAwareWriteSerializeMapOrStruct::Struct(
- SchemaAwareWriteSerializeStruct::new(self, record_schema),
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This also handles [`Schema::Union`].
+ fn checked_write_int(self, original_ty: &'static str, v: i32) -> Result<usize, Error> {
+ match self.schema {
+ Schema::Int | Schema::Date | Schema::TimeMillis => zig_i32(v, self.writer),
+ Schema::Union(union) => UnionSerializer::new(self.writer, union, self.config)
+ .checked_write_int(original_ty, v),
+ _ => Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis",
)),
- _ => Err(create_error(format!(
- "Expected Map or Union schema. Got: {schema}"
- ))),
}
}
- fn serialize_struct_with_schema<'a>(
- &'a mut self,
- name: &'static str,
- len: usize,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeStruct<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "struct",
- value: format!("{name}{{ ... }}. Cause: {cause}"),
- schema: schema.clone(),
- })
- };
-
- match schema {
- Schema::Record(record_schema) => {
- Ok(SchemaAwareWriteSerializeStruct::new(self, record_schema))
- }
- Schema::Ref { name: ref_name } => {
- let ref_schema = self.get_ref_schema(ref_name)?;
- self.serialize_struct_with_schema(name, len, ref_schema)
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This also handles [`Schema::Union`].
+ fn checked_write_long(self, original_ty: &'static str, v: i64) -> Result<usize, Error> {
+ match self.schema {
+ Schema::Long | Schema::TimeMicros | Schema::TimestampMillis | Schema::TimestampMicros
+ | Schema::TimestampNanos | Schema::LocalTimestampMillis | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos => {
+ zig_i64(v, self.writer)
}
- Schema::Union(union_schema) => {
- for (i, variant_schema) in union_schema.schemas.iter().enumerate() {
- match variant_schema {
- Schema::Record(inner)
- if inner.fields.len() == len && inner.name.name() == name =>
- {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_struct_with_schema(name, len, variant_schema);
- }
- Schema::Ref { name: _ } => {
- encode_int(i as i32, &mut *self.writer)?;
- return self.serialize_struct_with_schema(name, len, variant_schema);
- }
- _ => { /* skip */ }
- }
- }
- Err(create_error(format!(
- "Expected Record or Ref schema in {union_schema:?}"
- )))
+ Schema::Union(union) => UnionSerializer::new(self.writer, union, self.config).checked_write_long(original_ty, v),
+ _ => {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos}"))
}
- _ => Err(create_error(format!(
- "Expected Record, Ref or Union schema. Got: {schema}"
- ))),
}
}
- fn serialize_struct_variant_with_schema<'a>(
- &'a mut self,
- name: &'static str,
- variant_index: u32,
- variant: &'static str,
- len: usize,
- schema: &'s Schema,
- ) -> Result<SchemaAwareWriteSerializeStruct<'a, 's, W>, Error> {
- let create_error = |cause: String| {
- Error::new(Details::SerializeValueWithSchema {
- value_type: "struct variant",
- value: format!("{name}::{variant}{{ ... }} (size={len}. Cause: {cause})"),
- schema: schema.clone(),
- })
- };
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
- match schema {
- Schema::Union(union_schema) => {
- let variant_schema = union_schema
- .schemas
- .get(variant_index as usize)
- .ok_or_else(|| {
- create_error(format!(
- "Cannot find variant at position {variant_index} in {union_schema:?}"
- ))
- })?;
-
- encode_int(variant_index as i32, &mut self.writer)?;
- self.serialize_struct_with_schema(variant, len, variant_schema)
- }
- _ => Err(create_error(format!(
- "Expected Union schema. Got: {schema}"
- ))),
- }
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
}
}
-impl<'a, 's, W: Write> ser::Serializer for &'a mut SchemaAwareWriteSerializer<'s, W> {
+/// Indicate to the serializer that a record field default is being serialized.
+///
+/// This is needed because the serializer takes a `&'static str` for the enum name and variant name.
+/// When this value is encountered, the serializer will blindly trust the variant index.
+///
+/// To prevent users from abusing this fact, the string is compared by pointer value. Because the static
+/// is not public, there is no way for a user to obtain that value.
+static SERIALIZING_SCHEMA_DEFAULT: &str = "This value is compared by pointer value";
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for SchemaAwareSerializer<'s, 'w, W, S> {
+ /// The amount of bytes written.
type Ok = usize;
type Error = Error;
- type SerializeSeq = SchemaAwareWriteSerializeSeq<'a, 's, W>;
- type SerializeTuple = SchemaAwareWriteSerializeSeq<'a, 's, W>;
- type SerializeTupleStruct = SchemaAwareWriteSerializeTupleStruct<'a, 's, W>;
- type SerializeTupleVariant = SchemaAwareWriteSerializeTupleStruct<'a, 's, W>;
- type SerializeMap = SchemaAwareWriteSerializeMapOrStruct<'a, 's, W>;
- type SerializeStruct = SchemaAwareWriteSerializeStruct<'a, 's, W>;
- type SerializeStructVariant = SchemaAwareWriteSerializeStruct<'a, 's, W>;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
- fn serialize_bool(self, v: bool) -> Result<Self::Ok, Self::Error> {
- self.serialize_bool_with_schema(v, self.root_schema)
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Boolean => self.write_array([v as u8]),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_bool(v)
+ }
+ _ => Err(self.error("bool", "Expected Schema::Boolean")),
+ }
}
fn serialize_i8(self, v: i8) -> Result<Self::Ok, Self::Error> {
- self.serialize_i32(v as i32)
+ self.checked_write_int("i8", i32::from(v))
}
fn serialize_i16(self, v: i16) -> Result<Self::Ok, Self::Error> {
- self.serialize_i32(v as i32)
+ self.checked_write_int("i16", i32::from(v))
}
fn serialize_i32(self, v: i32) -> Result<Self::Ok, Self::Error> {
- self.serialize_i32_with_schema(v, self.root_schema)
+ self.checked_write_int("i32", v)
}
fn serialize_i64(self, v: i64) -> Result<Self::Ok, Self::Error> {
- self.serialize_i64_with_schema(v, self.root_schema)
+ self.checked_write_long("i64", v)
}
- fn serialize_i128(self, v: i128) -> Result<Self::Ok, Self::Error> {
- self.serialize_i128_with_schema(v, self.root_schema)
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "i128" => {
+ self.write_array(v.to_le_bytes())
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_i128(v)
+ }
+ _ => Err(self.error("i128", r#"Expected Schema::Fixed(name: "i128", size: 16)"#)),
+ }
}
fn serialize_u8(self, v: u8) -> Result<Self::Ok, Self::Error> {
- self.serialize_u8_with_schema(v, self.root_schema)
+ self.checked_write_int("u8", i32::from(v))
}
fn serialize_u16(self, v: u16) -> Result<Self::Ok, Self::Error> {
- self.serialize_u32(v as u32)
+ self.checked_write_int("u16", i32::from(v))
}
fn serialize_u32(self, v: u32) -> Result<Self::Ok, Self::Error> {
- self.serialize_u32_with_schema(v, self.root_schema)
+ self.checked_write_long("i64", i64::from(v))
}
- fn serialize_u64(self, v: u64) -> Result<Self::Ok, Self::Error> {
- self.serialize_u64_with_schema(v, self.root_schema)
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 8 && fixed.name.name() == "u64" => {
+ self.write_array(v.to_le_bytes())
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_u64(v)
+ }
+ _ => Err(self.error("u64", r#"Expected Schema::Fixed(name: "u64", size: 8)"#)),
+ }
}
- fn serialize_u128(self, v: u128) -> Result<Self::Ok, Self::Error> {
- self.serialize_u128_with_schema(v, self.root_schema)
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Fixed(fixed) if fixed.size == 16 && fixed.name.name() == "u128" => {
+ self.write_array(v.to_le_bytes())
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_u128(v)
+ }
+ _ => Err(self.error("u128", r#"Expected Schema::Fixed(name: "u128", size: 16)"#)),
+ }
}
- fn serialize_f32(self, v: f32) -> Result<Self::Ok, Self::Error> {
- self.serialize_f32_with_schema(v, self.root_schema)
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Float => self.write_array(v.to_le_bytes()),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_f32(v)
+ }
+ _ => Err(self.error("f32", "Expected Schema::Float")),
+ }
}
- fn serialize_f64(self, v: f64) -> Result<Self::Ok, Self::Error> {
- self.serialize_f64_with_schema(v, self.root_schema)
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Double => self.write_array(v.to_le_bytes()),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_f64(v)
+ }
+ _ => Err(self.error("f64", "Expected Schema::Double")),
+ }
}
- fn serialize_char(self, v: char) -> Result<Self::Ok, Self::Error> {
- self.serialize_char_with_schema(v, self.root_schema)
+ fn serialize_char(mut self, v: char) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ // Convert the UTF-32 character to UTF-8
+ Schema::String => self.write_bytes_with_len(v.to_string().as_bytes()),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_char(v)
+ }
+ _ => Err(self.error("char", "Expected Schema::String")),
+ }
}
- fn serialize_str(self, v: &str) -> Result<Self::Ok, Self::Error> {
- self.serialize_str_with_schema(v, self.root_schema)
+ fn serialize_str(mut self, v: &str) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::String | Schema::Uuid(UuidSchema::String) => {
+ self.write_bytes_with_len(v.as_bytes())
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_str(v)
+ }
+ _ => Err(self.error("string", "Expected Schema::String ")),
+ }
}
- fn serialize_bytes(self, v: &[u8]) -> Result<Self::Ok, Self::Error> {
- self.serialize_bytes_with_schema(v, self.root_schema)
+ fn serialize_bytes(mut self, v: &[u8]) -> Result<Self::Ok, Self::Error> {
+ match self.schema {
+ Schema::Bytes | Schema::BigDecimal | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Bytes, ..}) | Schema::Uuid(UuidSchema::Bytes) => {
+ self.write_bytes_with_len(v)
+ }
+ Schema::Fixed(fixed) | Schema::Decimal(DecimalSchema { inner: InnerDecimalSchema::Fixed(fixed), .. }) | Schema::Uuid(UuidSchema::Fixed(fixed)) | Schema::Duration(fixed) => {
+ if fixed.size != v.len() {
+ Err(self.error("bytes", format!("Fixed size ({}) does not match bytes length ({})", fixed.size, v.len())))
+ } else {
+ self.write_bytes(v)
+ }
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_bytes(v)
+ }
+ _ => Err(self.error("bytes", "Expected Schema::Bytes | Schema::Fixed | Schema::BigDecimal | Schema::Decimal | Schema::Uuid(Bytes | Fixed) | Schema::Duration")),
+ }
}
fn serialize_none(self) -> Result<Self::Ok, Self::Error> {
- self.serialize_none_with_schema(self.root_schema)
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && let Some(null_index) = union.index_of_schema_kind(SchemaKind::Null)
+ {
+ zig_i32(null_index as i32, &mut *self.writer)
+ } else {
+ Err(self.error("none", "Expected Schema::Union([Schema::Null, _])"))
+ }
}
fn serialize_some<T>(self, value: &T) -> Result<Self::Ok, Self::Error>
where
- T: ?Sized + ser::Serialize,
+ T: ?Sized + Serialize,
{
- self.serialize_some_with_schema(value, self.root_schema)
+ if let Schema::Union(union) = self.schema
+ && union.variants().len() == 2
+ && let Some(null_index) = union.index_of_schema_kind(SchemaKind::Null)
+ {
+ let some_index = (null_index + 1) & 1;
+ let mut bytes_written = zig_i32(some_index as i32, &mut *self.writer)?;
+ bytes_written +=
+ value.serialize(self.with_different_schema(&union.variants()[some_index])?)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("some", "Expected Schema::Union([Schema::Null, _])"))
+ }
}
fn serialize_unit(self) -> Result<Self::Ok, Self::Error> {
- self.serialize_none()
+ match self.schema {
+ Schema::Null => Ok(0),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_unit()
+ }
+ _ => Err(self.error("unit", "Expected Schema::Null")),
+ }
}
fn serialize_unit_struct(self, name: &'static str) -> Result<Self::Ok, Self::Error> {
- self.serialize_unit_struct_with_schema(name, self.root_schema)
+ match self.schema {
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ Ok(0)
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_unit_struct(name)
+ }
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields: [])"),
+ )),
+ }
}
fn serialize_unit_variant(
@@ -1948,7 +417,26 @@ impl<'a, 's, W: Write> ser::Serializer for &'a mut SchemaAwareWriteSerializer<'s
variant_index: u32,
variant: &'static str,
) -> Result<Self::Ok, Self::Error> {
- self.serialize_unit_variant_with_schema(name, variant_index, variant, self.root_schema)
+ match self.schema {
+ Schema::Enum(enum_schema) => {
+ // Plain enum
+ if name.as_ptr() == SERIALIZING_SCHEMA_DEFAULT.as_ptr() || enum_schema.symbols[variant_index as usize] == variant {
+ zig_i32(variant_index as i32, &mut *self.writer)
+ } else {
+ Err(self.error("unit variant", format!(r#"Expected symbol "{name}" at index {variant_index} in enum"#)))
+ }
+ }
+ Schema::Union(union) => match self.get_resolved_union_variant(union, variant_index)? {
+ // Bare union
+ Schema::Null => zig_i32(variant_index as i32, &mut *self.writer),
+ Schema::Record(record) if record.fields.is_empty() && record.name.name() == name => {
+ // Union of records
+ zig_i32(variant_index as i32, &mut *self.writer)
+ }
+ _ => Err(self.error("unit variant", format!("Expected Schema::Null | Schema::Record(name: {name}, fields: []) at index {variant_index} in the union"))),
+ }
+ _ => Err(self.error("unit variant", format!("Expected Schema::Enum(symbols[{variant_index}] == {name}) | Schema::Union(variants[{variant_index}] == Schema::Null | Schema::Record(name: {name}, fields: []))"))),
+ }
}
fn serialize_newtype_struct<T>(
@@ -1957,36 +445,90 @@ impl<'a, 's, W: Write> ser::Serializer for &'a mut SchemaAwareWriteSerializer<'s
value: &T,
) -> Result<Self::Ok, Self::Error>
where
- T: ?Sized + ser::Serialize,
+ T: ?Sized + Serialize,
{
- self.serialize_newtype_struct_with_schema(name, value, self.root_schema)
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == 1 && record.name.name() == name => {
+ let schema = &record.fields[0].schema;
+ value.serialize(self.with_different_schema(schema)?)
+ }
+ Schema::Union(union) => UnionSerializer::new(self.writer, union, self.config)
+ .serialize_newtype_struct(name, value),
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields: [_])"),
+ )),
+ }
}
fn serialize_newtype_variant<T>(
self,
- name: &'static str,
+ _name: &'static str,
variant_index: u32,
variant: &'static str,
value: &T,
) -> Result<Self::Ok, Self::Error>
where
- T: ?Sized + ser::Serialize,
+ T: ?Sized + Serialize,
{
- self.serialize_newtype_variant_with_schema(
- name,
- variant_index,
- variant,
- value,
- self.root_schema,
- )
+ match self.schema {
+ Schema::Union(union) => match self.get_resolved_union_variant(union, variant_index)? {
+ Schema::Record(record)
+ if record.fields.len() == 1
+ && record.name.name() == variant
+ && record
+ .attributes
+ .get("org.apache.avro.rust.union_of_records")
+ == Some(&Bool(true)) =>
+ {
+ // Union of records
+ let mut bytes_written = zig_i32(variant_index as i32, &mut *self.writer)?;
+ let schema = &record.fields[0].schema;
+ bytes_written += value.serialize(self.with_different_schema(schema)?)?;
+ Ok(bytes_written)
+ }
+ schema => {
+ let mut bytes_written = zig_i32(variant_index as i32, &mut *self.writer)?;
+ bytes_written += value.serialize(self.with_different_schema(schema)?)?;
+ Ok(bytes_written)
+ }
+ },
+ _ => Err(self.error("newtype variant", "Expected Schema::Union")),
+ }
}
fn serialize_seq(self, len: Option<usize>) -> Result<Self::SerializeSeq, Self::Error> {
- self.serialize_seq_with_schema(len, self.root_schema)
+ match self.schema {
+ Schema::Array(array) => {
+ BlockSerializer::array(self.writer, array, self.config, len, None)
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_seq(len)
+ }
+ _ => Err(self.error("seq", "Expected Schema::Array")),
+ }
}
fn serialize_tuple(self, len: usize) -> Result<Self::SerializeTuple, Self::Error> {
- self.serialize_tuple_with_schema(len, self.root_schema)
+ match self.schema {
+ Schema::Union(union) => {
+ // This needs to be matched first, otherwise the `schema if len == 1` will also
+ // match unions
+ UnionSerializer::new(self.writer, union, self.config).serialize_tuple(len)
+ }
+ // `len == 0` is not possible for derived Serialize implementations but users might use it.
+ // The derived Serialize implementations use `serialize_unit` instead
+ Schema::Null if len == 0 => Ok(TupleSerializer::unit(None)),
+ schema if len == 1 => Ok(TupleSerializer::one(self.writer, schema, self.config, None)),
+ Schema::Record(record) if record.fields.len() == len => Ok(TupleSerializer::many(
+ self.writer,
+ record,
+ self.config,
+ None,
+ )),
+ // This error case can only happen for len > 1
+ _ => Err(self.error("tuple", "Expected Schema::Record(fields.len() == {len}")),
+ }
}
fn serialize_tuple_struct(
@@ -1994,27 +536,74 @@ impl<'a, 's, W: Write> ser::Serializer for &'a mut SchemaAwareWriteSerializer<'s
name: &'static str,
len: usize,
) -> Result<Self::SerializeTupleStruct, Self::Error> {
- self.serialize_tuple_struct_with_schema(name, len, self.root_schema)
+ match self.schema {
+ Schema::Record(record) if record.fields.len() == len && record.name.name() == name => {
+ Ok(ManyTupleSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ None,
+ ))
+ }
+ Schema::Union(union) => UnionSerializer::new(self.writer, union, self.config)
+ .serialize_tuple_struct(name, len),
+ _ => Err(self.error(
+ "tuple struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == {len})"),
+ )),
+ }
}
fn serialize_tuple_variant(
self,
- name: &'static str,
+ _: &'static str,
variant_index: u32,
variant: &'static str,
len: usize,
) -> Result<Self::SerializeTupleVariant, Self::Error> {
- self.serialize_tuple_variant_with_schema(
- name,
- variant_index,
- variant,
- len,
- self.root_schema,
- )
+ if let Schema::Union(union) = self.schema
+ && let Schema::Record(record) = self.get_resolved_union_variant(union, variant_index)?
+ && record.fields.len() == len
+ && record.name.name() == variant
+ {
+ let bytes_written = zig_i32(variant_index as i32, &mut *self.writer)?;
+ Ok(ManyTupleSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ } else {
+ Err(self.error("tuple variant", format!("Expected Schema::Union(variants[{variant_index}] == Schema::Record(name: {variant}, fields.len() == {len}))")))
+ }
}
fn serialize_map(self, len: Option<usize>) -> Result<Self::SerializeMap, Self::Error> {
- self.serialize_map_with_schema(len, self.root_schema)
+ match self.schema {
+ Schema::Map(map) => Ok(MapOrRecordSerializer::map(
+ self.writer,
+ map,
+ self.config,
+ len,
+ None,
+ )?),
+ Schema::Record(record) => {
+ // Structs with flattened fields are serialized as a map
+ Ok(MapOrRecordSerializer::record(
+ self.writer,
+ record,
+ self.config,
+ len,
+ ))
+ }
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_map(len)
+ }
+ _ => Err(self.error(
+ "map",
+ "Expected Schema::Map | Schema::Record for structs with flattened fields",
+ )),
+ }
}
fn serialize_struct(
@@ -2022,27 +611,113 @@ impl<'a, 's, W: Write> ser::Serializer for &'a mut SchemaAwareWriteSerializer<'s
name: &'static str,
len: usize,
) -> Result<Self::SerializeStruct, Self::Error> {
- self.serialize_struct_with_schema(name, len, self.root_schema)
+ match self.schema {
+ // Serde is inconsistent with the `name` and `len` provided. When using internally tagged
+ // enums the name can be the name of the inner type of a newtype variant. The length can
+ // also change based on `serialize_if`.
+ Schema::Record(record) => Ok(RecordSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ None,
+ )),
+ Schema::Union(union) => {
+ UnionSerializer::new(self.writer, union, self.config).serialize_struct(name, len)
+ }
+ _ => Err(self.error("struct", "Expected Schema::Record")),
+ }
}
fn serialize_struct_variant(
self,
- name: &'static str,
+ _: &'static str,
variant_index: u32,
variant: &'static str,
len: usize,
) -> Result<Self::SerializeStructVariant, Self::Error> {
- self.serialize_struct_variant_with_schema(
- name,
- variant_index,
- variant,
- len,
- self.root_schema,
- )
+ if let Schema::Union(union) = self.schema
+ && let Schema::Record(record) = self.get_resolved_union_variant(union, variant_index)?
+ && record.fields.len() == len
+ && record.name.name() == variant
+ {
+ let bytes_written = zig_i32(variant_index as i32, &mut *self.writer)?;
+ Ok(RecordSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ } else {
+ Err(self.error("struct variant", format!("Expected Schema::Union(variants[{variant_index}] == Schema::Record(name: {variant}, fields.len() == {len}))")))
+ }
}
fn is_human_readable(&self) -> bool {
- crate::util::is_human_readable()
+ self.config.human_readable
+ }
+}
+
+pub enum MapOrRecordSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Map(BlockSerializer<'s, 'w, W, S>),
+ Record(RecordSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> MapOrRecordSerializer<'s, 'w, W, S> {
+ pub fn record(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::Record(RecordSerializer::new(writer, schema, config, bytes_written))
+ }
+
+ pub fn map(
+ writer: &'w mut W,
+ schema: &'s MapSchema,
+ config: Config<'s, S>,
+ len: Option<usize>,
+ bytes_written: Option<usize>,
+ ) -> Result<Self, Error> {
+ Ok(Self::Map(BlockSerializer::map(
+ writer,
+ schema,
+ config,
+ len,
+ bytes_written,
+ )?))
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for MapOrRecordSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ MapOrRecordSerializer::Map(map) => map.serialize_key(key),
+ MapOrRecordSerializer::Record(record) => record.serialize_key(key),
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ MapOrRecordSerializer::Map(map) => map.serialize_value(value),
+ MapOrRecordSerializer::Record(record) => record.serialize_value(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ MapOrRecordSerializer::Map(map) => map.end(),
+ MapOrRecordSerializer::Record(record) => record.end(),
+ }
}
}
@@ -2070,14 +745,25 @@ mod tests {
let schema = Schema::Null;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- ().serialize(&mut serializer)?;
- None::<()>.serialize(&mut serializer)?;
- None::<i32>.serialize(&mut serializer)?;
- None::<String>.serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ None::<()>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ None::<i32>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ None::<String>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), Vec::<u8>::new().as_slice());
@@ -2089,12 +775,23 @@ mod tests {
let schema = Schema::Boolean;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- true.serialize(&mut serializer)?;
- false.serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ true.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ false.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), &[1, 0]);
@@ -2106,16 +803,27 @@ mod tests {
let schema = Schema::Int;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 4u8.serialize(&mut serializer)?;
- 31u16.serialize(&mut serializer)?;
- 13u32.serialize(&mut serializer)?;
- 7i8.serialize(&mut serializer)?;
- (-57i16).serialize(&mut serializer)?;
- 129i32.serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ 4u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 31u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 13u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 7i8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ (-57i16).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 129i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), &[8, 62, 26, 14, 113, 130, 2]);
@@ -2127,18 +835,29 @@ mod tests {
let schema = Schema::Long;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
-
- 4u8.serialize(&mut serializer)?;
- 31u16.serialize(&mut serializer)?;
- 13u32.serialize(&mut serializer)?;
- 291u64.serialize(&mut serializer)?;
- 7i8.serialize(&mut serializer)?;
- (-57i16).serialize(&mut serializer)?;
- 129i32.serialize(&mut serializer)?;
- (-432i64).serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
+
+ 4u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 31u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 13u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 291u64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 7i8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ (-57i16).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 129i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ (-432i64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(
buffer.as_slice(),
@@ -2153,12 +872,23 @@ mod tests {
let schema = Schema::Float;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 4.7f32.serialize(&mut serializer)?;
- (-14.1f64).serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ 4.7f32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ (-14.1f64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), &[102, 102, 150, 64, 154, 153, 97, 193]);
@@ -2170,12 +900,23 @@ mod tests {
let schema = Schema::Float;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 4.7f32.serialize(&mut serializer)?;
- (-14.1f64).serialize(&mut serializer)?;
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ 4.7f32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ (-14.1f64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), &[102, 102, 150, 64, 154, 153, 97, 193]);
@@ -2187,13 +928,28 @@ mod tests {
let schema = Schema::Bytes;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 'a'.serialize(&mut serializer)?;
- "test".serialize(&mut serializer)?;
- Bytes::new(&[12, 3, 7, 91, 4]).serialize(&mut serializer)?;
- assert!(().serialize(&mut serializer).is_err());
- assert!(PhantomData::<String>.serialize(&mut serializer).is_err());
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ "test".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ Bytes::new(&[12, 3, 7, 91, 4]).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
+ assert!(
+ ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ PhantomData::<String>
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(
buffer.as_slice(),
@@ -2208,13 +964,28 @@ mod tests {
let schema = Schema::String;
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 'a'.serialize(&mut serializer)?;
- "test".serialize(&mut serializer)?;
- Bytes::new(&[12, 3, 7, 91, 4]).serialize(&mut serializer)?;
- assert!(().serialize(&mut serializer).is_err());
- assert!(PhantomData::<String>.serialize(&mut serializer).is_err());
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ "test".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ Bytes::new(&[12, 3, 7, 91, 4]).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
+ assert!(
+ ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ PhantomData::<String>
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(
buffer.as_slice(),
@@ -2253,22 +1024,37 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let good_record = GoodTestRecord {
string_field: String::from("test"),
int_field: 10,
};
- good_record.serialize(&mut serializer)?;
+ good_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let bad_record = BadTestRecord {
foo_string_field: String::from("test"),
bar_int_field: 10,
};
- assert!(bad_record.serialize(&mut serializer).is_err());
+ assert!(
+ bad_record
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
- assert!("".serialize(&mut serializer).is_err());
- assert!(Some("").serialize(&mut serializer).is_err());
+ assert!(
+ "".serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
+ assert!(
+ Some("")
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .is_err()
+ );
assert_eq!(buffer.as_slice(), &[8, b't', b'e', b's', b't', 20]);
@@ -2287,11 +1073,15 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
#[derive(Serialize)]
struct EmptyRecord;
- EmptyRecord.serialize(&mut serializer)?;
+ EmptyRecord.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
#[derive(Serialize)]
struct NonEmptyRecord {
@@ -2301,14 +1091,14 @@ mod tests {
foo: "bar".to_string(),
};
match record
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::FieldName(field_name)) if field_name == "foo" => (),
unexpected => panic!("Expected an error. Got: {unexpected:?}"),
}
- match ().serialize(&mut serializer).map_err(Error::into_details) {
+ match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
Err(Details::SerializeValueWithSchema {
value_type,
value,
@@ -2346,14 +1136,18 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- Suit::Spades.serialize(&mut serializer)?;
- Suit::Hearts.serialize(&mut serializer)?;
- Suit::Diamonds.serialize(&mut serializer)?;
- Suit::Clubs.serialize(&mut serializer)?;
+ Suit::Spades.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ Suit::Hearts.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ Suit::Diamonds.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ Suit::Clubs.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match None::<()>
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2384,13 +1178,17 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let arr: Vec<i64> = vec![10, 5, 400];
- arr.serialize(&mut serializer)?;
+ arr.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match vec![1_f32]
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2421,17 +1219,24 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let mut map: BTreeMap<String, i64> = BTreeMap::new();
map.insert(String::from("item1"), 10);
map.insert(String::from("item2"), 400);
- map.serialize(&mut serializer)?;
+ map.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let mut map: BTreeMap<String, &str> = BTreeMap::new();
map.insert(String::from("item1"), "value1");
- match map.serialize(&mut serializer).map_err(Error::into_details) {
+ match map
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .map_err(Error::into_details)
+ {
Err(Details::SerializeValueWithSchema {
value_type,
value,
@@ -2471,15 +1276,23 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- Some(10i64).serialize(&mut serializer)?;
- None::<i64>.serialize(&mut serializer)?;
- NullableLong::Long(400).serialize(&mut serializer)?;
- NullableLong::Null.serialize(&mut serializer)?;
+ Some(10i64).serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ None::<i64>.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ NullableLong::Long(400).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
+ NullableLong::Null.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match "invalid"
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2519,14 +1332,26 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- LongOrString::Null.serialize(&mut serializer)?;
- LongOrString::Long(400).serialize(&mut serializer)?;
- LongOrString::Str(String::from("test")).serialize(&mut serializer)?;
+ LongOrString::Null.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ LongOrString::Long(400).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
+ LongOrString::Str(String::from("test")).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
match 1_f64
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2564,13 +1389,21 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- Bytes::new(&[10, 124, 31, 97, 14, 201, 3, 88]).serialize(&mut serializer)?;
+ Bytes::new(&[10, 124, 31, 97, 14, 201, 3, 88]).serialize(SchemaAwareSerializer::new(
+ &mut buffer,
+ &schema,
+ config,
+ )?)?;
// non-8 size
match Bytes::new(&[123])
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2590,7 +1423,7 @@ mod tests {
// array
match [1; 8]
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2607,7 +1440,7 @@ mod tests {
// slice
match &[1, 2, 3, 4, 5, 6, 7, 8]
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2640,12 +1473,16 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let val = Decimal::from(&[251, 155]);
- val.serialize(&mut serializer)?;
+ val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- match ().serialize(&mut serializer).map_err(Error::into_details) {
+ match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
Err(Details::SerializeValueWithSchema {
value_type,
value,
@@ -2678,12 +1515,16 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let val = Decimal::from(&[0, 0, 0, 0, 0, 0, 251, 155]);
- val.serialize(&mut serializer)?;
+ val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- match ().serialize(&mut serializer).map_err(Error::into_details) {
+ match ().serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?).map_err(Error::into_details) {
Err(Details::SerializeValueWithSchema {
value_type,
value,
@@ -2712,10 +1553,14 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let val = BigDecimal::new(BigInt::new(Sign::Plus, vec![50024]), 2);
- val.serialize(&mut serializer)?;
+ val.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[10, 6, 0, 195, 104, 4]);
@@ -2736,13 +1581,20 @@ mod tests {
assert!(!crate::util::is_human_readable());
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
"8c28da81-238c-4326-bddd-4e3d00cc5099"
.parse::<Uuid>()?
- .serialize(&mut serializer)?;
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
- match 1_u8.serialize(&mut serializer).map_err(Error::into_details) {
+ match 1_u8
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .map_err(Error::into_details)
+ {
Err(Details::SerializeValueWithSchema {
value_type,
value,
@@ -2776,16 +1628,20 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 100_u8.serialize(&mut serializer)?;
- 1000_u16.serialize(&mut serializer)?;
- 10000_u32.serialize(&mut serializer)?;
- 1000_i16.serialize(&mut serializer)?;
- 10000_i32.serialize(&mut serializer)?;
+ 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match 10000_f32
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2819,16 +1675,20 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 100_u8.serialize(&mut serializer)?;
- 1000_u16.serialize(&mut serializer)?;
- 10000_u32.serialize(&mut serializer)?;
- 1000_i16.serialize(&mut serializer)?;
- 10000_i32.serialize(&mut serializer)?;
+ 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match 10000_f32
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2862,17 +1722,21 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 100_u8.serialize(&mut serializer)?;
- 1000_u16.serialize(&mut serializer)?;
- 10000_u32.serialize(&mut serializer)?;
- 1000_i16.serialize(&mut serializer)?;
- 10000_i32.serialize(&mut serializer)?;
- 10000_i64.serialize(&mut serializer)?;
+ 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match 10000_f32
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2909,18 +1773,20 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer =
- SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
-
- 100_u8.serialize(&mut serializer)?;
- 1000_u16.serialize(&mut serializer)?;
- 10000_u32.serialize(&mut serializer)?;
- 1000_i16.serialize(&mut serializer)?;
- 10000_i32.serialize(&mut serializer)?;
- 10000_i64.serialize(&mut serializer)?;
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
+ 100_u8.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_u16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_u32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 1000_i16.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i32.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
+ 10000_i64.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match 10000_f64
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -2968,14 +1834,18 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
let duration_bytes =
ByteArray::new(Duration::new(Months::new(3), Days::new(2), Millis::new(1200)).into());
- duration_bytes.serialize(&mut serializer)?;
+ duration_bytes.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
match [1; 12]
- .serialize(&mut serializer)
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
.map_err(Error::into_details)
{
Err(Details::SerializeValueWithSchema {
@@ -3028,8 +1898,11 @@ mod tests {
assert!(!crate::util::is_human_readable());
let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let mut serializer =
- SchemaAwareWriteSerializer::new(&mut buffer, &schema, rs.get_names(), None);
+ let config = Config {
+ names: rs.get_names(),
+ target_block_size: None,
+ human_readable: false,
+ };
let good_record = TestRecord {
string_field: String::from("test"),
@@ -3044,7 +1917,7 @@ mod tests {
inner_record: None,
})),
};
- good_record.serialize(&mut serializer)?;
+ good_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(
buffer.as_slice(),
@@ -3109,29 +1982,32 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let mut serializer =
- SchemaAwareWriteSerializer::new(&mut buffer, &schema, rs.get_names(), None);
+ let config = Config {
+ names: rs.get_names(),
+ target_block_size: None,
+ human_readable: false,
+ };
let foo_record = TestRecord {
inner_union: InnerUnion::InnerVariantFoo(InnerRecordFoo {
foo: String::from("foo"),
}),
};
- foo_record.serialize(&mut serializer)?;
+ foo_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let bar_record = TestRecord {
inner_union: InnerUnion::InnerVariantBar(InnerRecordBar {
bar: String::from("bar"),
}),
};
- bar_record.serialize(&mut serializer)?;
+ bar_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let int_record = TestRecord {
inner_union: InnerUnion::IntField(1),
};
- int_record.serialize(&mut serializer)?;
+ int_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let string_record = TestRecord {
inner_union: InnerUnion::StringField(String::from("string")),
};
- string_record.serialize(&mut serializer)?;
+ string_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
Ok(())
}
@@ -3186,31 +2062,34 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let rs = ResolvedSchema::try_from(&schema)?;
- let mut serializer =
- SchemaAwareWriteSerializer::new(&mut buffer, &schema, rs.get_names(), None);
+ let config = Config {
+ names: rs.get_names(),
+ target_block_size: None,
+ human_readable: false,
+ };
let null_record = TestRecord { inner_union: None };
- null_record.serialize(&mut serializer)?;
+ null_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let foo_record = TestRecord {
inner_union: Some(InnerUnion::InnerVariantFoo(InnerRecordFoo {
foo: String::from("foo"),
})),
};
- foo_record.serialize(&mut serializer)?;
+ foo_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let bar_record = TestRecord {
inner_union: Some(InnerUnion::InnerVariantBar(InnerRecordBar {
bar: String::from("bar"),
})),
};
- bar_record.serialize(&mut serializer)?;
+ bar_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let int_record = TestRecord {
inner_union: Some(InnerUnion::IntField(1)),
};
- int_record.serialize(&mut serializer)?;
+ int_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
let string_record = TestRecord {
inner_union: Some(InnerUnion::StringField(String::from("string"))),
};
- string_record.serialize(&mut serializer)?;
+ string_record.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
Ok(())
}
@@ -3287,9 +2166,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 'a'.serialize(&mut serializer)?;
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[2, b'a']);
@@ -3302,9 +2185,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 'a'.serialize(&mut serializer)?;
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[2, b'a']);
@@ -3323,9 +2210,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- 'a'.serialize(&mut serializer)?;
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[b'a', 0, 0, 0]);
@@ -3338,9 +2229,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- '👹'.serialize(&mut serializer)?;
+ '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[8, 240, 159, 145, 185]);
@@ -3353,9 +2248,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- '👹'.serialize(&mut serializer)?;
+ '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(buffer.as_slice(), &[8, 240, 159, 145, 185]);
@@ -3374,9 +2273,13 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- '👹'.serialize(&mut serializer)?;
+ '👹'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
// This is a different byte value than the tests above. This is because by creating a String
// the unicode value is normalized by Rust
@@ -3397,10 +2300,16 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- 'a'.serialize(&mut serializer).unwrap_err().details(),
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3419,10 +2328,16 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- 'a'.serialize(&mut serializer).unwrap_err().details(),
+ 'a'.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3441,9 +2356,14 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- let bytes_written = i128::MAX.serialize(&mut serializer)?;
+ let bytes_written =
+ i128::MAX.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(bytes_written, 16);
assert_eq!(
@@ -3469,10 +2389,17 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- i128::MAX.serialize(&mut serializer).unwrap_err().details(),
+ i128::MAX
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3491,10 +2418,17 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- i128::MAX.serialize(&mut serializer).unwrap_err().details(),
+ i128::MAX
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3513,9 +2447,14 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
- let bytes_written = u128::MAX.serialize(&mut serializer)?;
+ let bytes_written =
+ u128::MAX.serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)?;
assert_eq!(bytes_written, 16);
assert_eq!(
@@ -3541,10 +2480,17 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- u128::MAX.serialize(&mut serializer).unwrap_err().details(),
+ u128::MAX
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3563,10 +2509,17 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
assert!(matches!(
- u128::MAX.serialize(&mut serializer).unwrap_err().details(),
+ u128::MAX
+ .serialize(SchemaAwareSerializer::new(&mut buffer, &schema, config)?)
+ .unwrap_err()
+ .details(),
Details::SerializeValueWithSchema { .. }
));
@@ -3585,11 +2538,20 @@ mod tests {
let mut buffer: Vec<u8> = Vec::new();
let names = HashMap::new();
- let mut serializer = SchemaAwareWriteSerializer::new(&mut buffer, &schema, &names, None);
- let bytes_written = crate::serde::fixed::serialize(&[0, 1, 2, 3], &mut serializer)?;
+ let config = Config::<'_, Schema> {
+ names: &names,
+ target_block_size: None,
+ human_readable: false,
+ };
+ let bytes_written = crate::serde::fixed::serialize(
+ &[0, 1, 2, 3],
+ SchemaAwareSerializer::new(&mut buffer, &schema, config)?,
+ )?;
assert_eq!(bytes_written, 4);
- let bytes_written =
- crate::serde::fixed::serialize(&[4, 5, 6, 7, 8, 9, 10, 11], &mut serializer)?;
+ let bytes_written = crate::serde::fixed::serialize(
+ &[4, 5, 6, 7, 8, 9, 10, 11],
+ SchemaAwareSerializer::new(&mut buffer, &schema, config)?,
+ )?;
assert_eq!(bytes_written, 8);
assert_eq!(buffer, &[0, 0, 1, 2, 3, 2, 4, 5, 6, 7, 8, 9, 10, 11][..]);
diff --git a/avro/src/serde/ser_schema/record/field_default.rs b/avro/src/serde/ser_schema/record/field_default.rs
new file mode 100644
index 0000000..14716e8
--- /dev/null
+++ b/avro/src/serde/ser_schema/record/field_default.rs
@@ -0,0 +1,199 @@
+use crate::Schema;
+use crate::schema::{SchemaKind, UnionSchema, UuidSchema};
+use crate::serde::ser_schema::SERIALIZING_SCHEMA_DEFAULT;
+use serde::{Serialize, Serializer, ser::Error};
+use serde_json::Value;
+
+pub struct SchemaAwareRecordFieldDefault<'v, 's> {
+ value: &'v Value,
+ schema: &'s Schema,
+}
+
+impl<'v, 's> SchemaAwareRecordFieldDefault<'v, 's> {
+ pub fn new(value: &'v Value, schema: &'s Schema) -> Self {
+ SchemaAwareRecordFieldDefault { value, schema }
+ }
+
+ fn serialize_as_newtype_variant<S: Serializer>(
+ &self,
+ serializer: S,
+ index: usize,
+ union: &'s UnionSchema,
+ ) -> Result<S::Ok, S::Error> {
+ let value = Self::new(self.value, &union.variants()[index]);
+ serializer.serialize_newtype_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ &value,
+ )
+ }
+
+ fn recursive_type_check(value: &Value, schema: &'s Schema) -> bool {
+ match (value, schema) {
+ (Value::Null, Schema::Null)
+ | (Value::Bool(_), Schema::Boolean)
+ | (Value::String(_), Schema::Bytes | Schema::String) => true,
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ i32::try_from(long).is_ok()
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => true,
+ (Value::Number(n), Schema::Float | Schema::Double) if n.is_f64() => true,
+ (Value::String(s), Schema::Fixed(fixed)) => s.len() == fixed.size,
+ (Value::String(s), Schema::Enum(enum_schema)) => enum_schema.symbols.contains(s),
+ (Value::Object(o), Schema::Record(record)) => record.fields.iter().all(|field| {
+ if let Some(value) = o.get(&field.name) {
+ Self::recursive_type_check(value, &field.schema)
+ } else {
+ field.default.is_some()
+ }
+ }),
+ (Value::Object(o), Schema::Map(map)) => o
+ .values()
+ .all(|value| Self::recursive_type_check(value, &map.types)),
+ (Value::Array(a), Schema::Array(array)) => a
+ .iter()
+ .all(|value| Self::recursive_type_check(value, &array.items)),
+ (_, Schema::Union(union)) => union
+ .variants()
+ .iter()
+ .any(|variant| Self::recursive_type_check(value, variant)),
+ _ => false,
+ }
+ }
+}
+
+impl<'v, 's> Serialize for SchemaAwareRecordFieldDefault<'v, 's> {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match (&self.value, self.schema) {
+ (Value::Null, Schema::Null) => serializer.serialize_unit(),
+ (Value::Bool(boolean), Schema::Boolean) => serializer.serialize_bool(*boolean),
+ (Value::Number(n), Schema::Int | Schema::Date | Schema::TimeMillis) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ let int = i32::try_from(long).map_err(|_| {
+ S::Error::custom(format!("Default {long} is too large for {:?}", self.schema))
+ })?;
+ serializer.serialize_i32(int)
+ }
+ (
+ Value::Number(n),
+ Schema::Long
+ | Schema::TimeMicros
+ | Schema::TimestampMillis
+ | Schema::TimestampMicros
+ | Schema::TimestampNanos
+ | Schema::LocalTimestampMillis
+ | Schema::LocalTimestampMicros
+ | Schema::LocalTimestampNanos,
+ ) if n.is_i64() => {
+ let long = n.as_i64().unwrap();
+ serializer.serialize_i64(long)
+ }
+ (Value::Number(n), Schema::Float) if n.is_f64() => {
+ let float = n.as_f64().unwrap();
+ serializer.serialize_f32(float as f32)
+ }
+ (Value::Number(n), Schema::Double) if n.is_f64() => {
+ let double = n.as_f64().unwrap();
+ serializer.serialize_f64(double)
+ }
+ (
+ Value::String(s),
+ Schema::Bytes
+ | Schema::Fixed(_)
+ | Schema::Uuid(UuidSchema::Bytes | UuidSchema::Fixed(_))
+ | Schema::BigDecimal
+ | Schema::Decimal(_)
+ | Schema::Duration(_)
+ | Schema::Date,
+ ) => serializer.serialize_bytes(s.as_bytes()),
+ (Value::String(s), Schema::String | Schema::Uuid(UuidSchema::String)) => {
+ serializer.serialize_str(s)
+ }
+ (Value::String(s), Schema::Enum(enum_schema)) => {
+ let Some((variant_index, _)) = enum_schema
+ .symbols
+ .iter()
+ .enumerate()
+ .find(|(_i, symbol)| *symbol == s)
+ else {
+ return Err(S::Error::custom(format!(
+ "Could not find `{s}` in enum: {enum_schema:?}"
+ )));
+ };
+
+ serializer.serialize_unit_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ variant_index as u32,
+ SERIALIZING_SCHEMA_DEFAULT,
+ )
+ }
+ // This abuses the support for flattened fields, which are also serialized as a map.
+ (Value::Object(o), Schema::Record(record)) => serializer.collect_map(
+ o.iter()
+ .enumerate()
+ .map(|(i, (k, v))| (k, Self::new(v, &record.fields[i].schema))),
+ ),
+ (Value::Object(o), Schema::Map(map)) => {
+ serializer.collect_map(o.iter().map(|(k, v)| (k, Self::new(v, &map.types))))
+ }
+ (Value::Array(a), Schema::Array(array)) => {
+ serializer.collect_seq(a.iter().map(|v| Self::new(v, &array.items)))
+ }
+ (_, Schema::Union(union)) => {
+ if union.variants().len() == 2
+ && let Some(null_index) = union.index_of_schema_kind(SchemaKind::Null)
+ {
+ // Fast path for options
+ if self.value == &Value::Null {
+ serializer.serialize_none()
+ } else {
+ let some_index = (null_index + 1) & 1;
+ let value = Self::new(self.value, &union.variants()[some_index]);
+ serializer.serialize_some(&value)
+ }
+ } else {
+ // Find the first variant that can match this value
+ for (index, variant) in union.variants().iter().enumerate() {
+ match (self.value, variant) {
+ (Value::Null, Schema::Null) => {
+ let index = index as u32;
+ return serializer.serialize_unit_variant(
+ SERIALIZING_SCHEMA_DEFAULT,
+ index,
+ SERIALIZING_SCHEMA_DEFAULT,
+ );
+ }
+ _ if Self::recursive_type_check(self.value, variant) => {
+ return self.serialize_as_newtype_variant(serializer, index, union);
+ }
+ _ => {}
+ }
+ }
+ Err(S::Error::custom(format!(
+ "Could not match default to any variant of {:?}, default: {:?}",
+ self.schema, self.value
+ )))
+ }
+ }
+ _ => Err(S::Error::custom(format!(
+ "Unexpected default for {:?}, default: {:?}",
+ self.schema, self.value
+ ))),
+ }
+ }
+}
diff --git a/avro/src/serde/ser_schema/record/mod.rs b/avro/src/serde/ser_schema/record/mod.rs
new file mode 100644
index 0000000..853867d
--- /dev/null
+++ b/avro/src/serde/ser_schema/record/mod.rs
@@ -0,0 +1,191 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod field_default;
+
+use super::{Config, SchemaAwareSerializer};
+use crate::error::Details;
+use crate::schema::RecordSchema;
+use crate::serde::ser_schema::record::field_default::SchemaAwareRecordFieldDefault;
+use crate::serde::util::StringSerializer;
+use crate::{Error, Schema};
+use serde::Serialize;
+use serde::ser::{SerializeMap, SerializeStruct, SerializeStructVariant};
+use std::borrow::Borrow;
+use std::io::Write;
+
+pub struct RecordSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ /// The field that should be written now.
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> RecordSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ record: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ record,
+ config,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+
+ fn serialize_next_field<T: ?Sized + Serialize>(
+ &mut self,
+ position: usize,
+ value: &T,
+ ) -> Result<(), Error> {
+ // Serialize any skipped fields using their default value
+ while self.field_position < position {
+ self.serialize_default(position)?;
+ }
+ let field = &self.record.fields[position];
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &field.schema,
+ self.config,
+ )?)?;
+ self.field_position += 1;
+
+ Ok(())
+ }
+
+ fn serialize_default(&mut self, position: usize) -> Result<(), Error> {
+ let field = &self.record.fields[position];
+ if let Some(default) = &field.default {
+ self.serialize_next_field(
+ self.field_position,
+ &SchemaAwareRecordFieldDefault::new(default, &field.schema),
+ )
+ .map_err(|e| {
+ Details::SerializeRecordFieldWithSchema {
+ field_name: field.name.clone(),
+ record_schema: Schema::Record(self.record.clone()),
+ error: Box::new(e),
+ }
+ .into()
+ })
+ } else {
+ Err(Details::MissingDefaultForSkippedField {
+ field_name: field.name.clone(),
+ schema: Schema::Record(self.record.clone()),
+ }
+ .into())
+ }
+ }
+
+ fn end(mut self) -> Result<usize, Error> {
+ // Write any fields that were skipped by `#[serde(skip)]` or `#[serde(skip_serializing{,_if}]`
+ while self.field_position != self.record.fields.len() {
+ self.serialize_default(self.field_position)?;
+ }
+
+ Ok(self.bytes_written)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeStruct for RecordSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ if let Some(position) = self.record.lookup.get(key).copied() {
+ self.serialize_next_field(position, value)
+ } else {
+ Err(Details::GetField(key.to_string()).into())
+ }
+ }
+
+ fn skip_field(&mut self, key: &'static str) -> Result<(), Self::Error> {
+ if let Some(position) = self.record.lookup.get(key).copied() {
+ self.serialize_default(position)
+ } else {
+ Err(Details::GetField(key.to_string()).into())
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeMap for RecordSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_key<T>(&mut self, key: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let name = key.serialize(StringSerializer)?;
+ if let Some(position) = self.record.lookup.get(&name).copied() {
+ // Write any skipped fields now, so that `serialize_value` can just call `serialize_next_field`
+ while self.field_position < position {
+ self.serialize_default(self.field_position)?;
+ }
+ Ok(())
+ } else {
+ Err(Details::FieldName(name.to_string()).into())
+ }
+ }
+
+ fn serialize_value<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ // `serialize_key` made sure that all skipped fields have been written
+ self.serialize_next_field(self.field_position, value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeStructVariant
+ for RecordSerializer<'s, 'w, W, S>
+{
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_field<T>(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ <Self as SerializeStruct>::serialize_field(self, key, value)
+ }
+
+ fn skip_field(&mut self, key: &'static str) -> Result<(), Self::Error> {
+ <Self as SerializeStruct>::skip_field(self, key)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ self.end()
+ }
+}
diff --git a/avro/src/serde/ser_schema/tuple.rs b/avro/src/serde/ser_schema/tuple.rs
new file mode 100644
index 0000000..7aa2a79
--- /dev/null
+++ b/avro/src/serde/ser_schema/tuple.rs
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{Config, SchemaAwareSerializer, union::UnionSerializer};
+use crate::error::Details;
+use crate::schema::{RecordSchema, UnionSchema};
+use crate::{Error, Schema};
+use serde::Serialize;
+use serde::ser::{SerializeTuple, SerializeTupleStruct, SerializeTupleVariant};
+use std::borrow::Borrow;
+use std::io::Write;
+
+#[expect(
+ private_interfaces,
+ reason = "One{,Union}TupleSerializer should not be used directly"
+)]
+pub enum TupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ Unit(usize),
+ One(OneTupleSerializer<'s, 'w, W, S>),
+ /// This exists because we can't create a `&Schema::Union` from a `&UnionSchema`
+ OneUnion(OneUnionTupleSerializer<'s, 'w, W, S>),
+ Many(ManyTupleSerializer<'s, 'w, W, S>),
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> TupleSerializer<'s, 'w, W, S> {
+ pub fn unit(bytes_written: Option<usize>) -> Self {
+ Self::Unit(bytes_written.unwrap_or(0))
+ }
+
+ pub fn one(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::One(OneTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+
+ pub fn one_union(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self::OneUnion(OneUnionTupleSerializer::new(writer, union, config))
+ }
+
+ pub fn many(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self::Many(ManyTupleSerializer::new(
+ writer,
+ schema,
+ config,
+ bytes_written,
+ ))
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for TupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self {
+ TupleSerializer::Unit(_) => Err(Error::new(Details::SerializeValueWithSchema {
+ value_type: "tuple",
+ value: "Expected no elements for the unit tuple".into(),
+ schema: Schema::Null,
+ })),
+ TupleSerializer::One(one) => one.serialize_element(value),
+ TupleSerializer::OneUnion(one) => one.serialize_element(value),
+ TupleSerializer::Many(many) => many.serialize_element(value),
+ }
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ match self {
+ TupleSerializer::Unit(bytes_written) => Ok(bytes_written),
+ TupleSerializer::One(one) => one.end(),
+ TupleSerializer::OneUnion(one) => one.end(),
+ TupleSerializer::Many(many) => SerializeTuple::end(many),
+ }
+ }
+}
+
+pub struct ManyTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ field_position: usize,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> ManyTupleSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s RecordSchema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ schema,
+ config,
+ field_position: 0,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for ManyTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ let schema = &self.schema.fields[self.field_position].schema;
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ schema,
+ self.config,
+ )?)?;
+ self.field_position += 1;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ assert_eq!(self.field_position, self.schema.fields.len());
+ Ok(self.bytes_written)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTupleStruct
+ for ManyTupleSerializer<'s, 'w, W, S>
+{
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_field<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ SerializeTuple::serialize_element(self, value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ SerializeTuple::end(self)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTupleVariant
+ for ManyTupleSerializer<'s, 'w, W, S>
+{
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_field<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ SerializeTuple::serialize_element(self, value)
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ SerializeTuple::end(self)
+ }
+}
+
+struct OneTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> OneTupleSerializer<'s, 'w, W, S> {
+ pub fn new(
+ writer: &'w mut W,
+ schema: &'s Schema,
+ config: Config<'s, S>,
+ bytes_written: Option<usize>,
+ ) -> Self {
+ Self {
+ writer,
+ schema,
+ config,
+ bytes_written: bytes_written.unwrap_or(0),
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for OneTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self.schema {
+ Schema::Union(union) => {
+ self.bytes_written +=
+ value.serialize(UnionSerializer::new(self.writer, union, self.config))?;
+ }
+ schema => {
+ self.bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ schema,
+ self.config,
+ )?)?;
+ }
+ }
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ Ok(self.bytes_written)
+ }
+}
+
+struct OneUnionTupleSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+ bytes_written: usize,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> OneUnionTupleSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ Self {
+ writer,
+ union,
+ config,
+ bytes_written: 0,
+ }
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> SerializeTuple for OneUnionTupleSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+
+ fn serialize_element<T>(&mut self, value: &T) -> Result<(), Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ self.bytes_written +=
+ value.serialize(UnionSerializer::new(self.writer, self.union, self.config))?;
+ Ok(())
+ }
+
+ fn end(self) -> Result<Self::Ok, Self::Error> {
+ Ok(self.bytes_written)
+ }
+}
diff --git a/avro/src/serde/ser_schema/union.rs b/avro/src/serde/ser_schema/union.rs
new file mode 100644
index 0000000..a893fb5
--- /dev/null
+++ b/avro/src/serde/ser_schema/union.rs
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{Config, MapOrRecordSerializer, SchemaAwareSerializer};
+use crate::error::Details;
+use crate::schema::{FixedSchema, SchemaKind, UnionSchema};
+use crate::serde::ser_schema::block::BlockSerializer;
+use crate::serde::ser_schema::record::RecordSerializer;
+use crate::serde::ser_schema::tuple::{ManyTupleSerializer, TupleSerializer};
+use crate::serde::with::{BytesType, SER_BYTES_TYPE};
+use crate::util::{zig_i32, zig_i64};
+use crate::{Error, Schema};
+use serde::{Serialize, Serializer};
+use std::borrow::Borrow;
+use std::io::Write;
+
+pub struct UnionSerializer<'s, 'w, W: Write, S: Borrow<Schema>> {
+ writer: &'w mut W,
+ union: &'s UnionSchema,
+ config: Config<'s, S>,
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> UnionSerializer<'s, 'w, W, S> {
+ pub fn new(writer: &'w mut W, union: &'s UnionSchema, config: Config<'s, S>) -> Self {
+ UnionSerializer {
+ writer,
+ union,
+ config,
+ }
+ }
+
+ fn error(&self, ty: &'static str, error: impl Into<String>) -> Error {
+ Error::new(Details::SerializeValueWithSchema {
+ value_type: ty,
+ value: error.into(),
+ schema: Schema::Union(self.union.clone()),
+ })
+ }
+
+ /// Write an integer to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Int`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_int(
+ &mut self,
+ original_ty: &'static str,
+ v: i32,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Int) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(
+ original_ty,
+ "Expected Schema::Int | Schema::Date | Schema::TimeMillis in variants",
+ ))?
+ }
+ }
+
+ /// Write a long to the writer.
+ ///
+ /// This will check that the current schema is [`Schema::Long`] or a logical type based on that.
+ /// This will write the union index.
+ pub(super) fn checked_write_long(
+ &mut self,
+ original_ty: &'static str,
+ v: i64,
+ ) -> Result<usize, Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Long) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i64(v, &mut *self.writer)?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error(original_ty, "Expected Schema::Long | Schema::TimeMicros | Schema::{,Local}Timestamp{Millis,Micros,Nanos} in variants"))
+ }
+ }
+
+ /// Write bytes to the writer with preceding length header.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes_with_len(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ let mut bytes_written = 0;
+ bytes_written += zig_i64(bytes.len() as i64, &mut *self.writer)?;
+ bytes_written += self.write_bytes(bytes)?;
+ Ok(bytes_written)
+ }
+
+ /// Write bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_bytes(&mut self, bytes: &[u8]) -> Result<usize, Error> {
+ self.writer.write_all(bytes).map_err(Details::WriteBytes)?;
+ Ok(bytes.len())
+ }
+
+ /// Write an array of `n` bytes to the writer.
+ ///
+ /// This does not check the current schema and does not write the union index.
+ fn write_array<const N: usize>(&mut self, bytes: [u8; N]) -> Result<usize, Error> {
+ self.write_bytes(&bytes)?;
+ Ok(N)
+ }
+}
+
+impl<'s, 'w, W: Write, S: Borrow<Schema>> Serializer for UnionSerializer<'s, 'w, W, S> {
+ type Ok = usize;
+ type Error = Error;
+ type SerializeSeq = BlockSerializer<'s, 'w, W, S>;
+ type SerializeTuple = TupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleStruct = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeTupleVariant = ManyTupleSerializer<'s, 'w, W, S>;
+ type SerializeMap = MapOrRecordSerializer<'s, 'w, W, S>;
+ type SerializeStruct = RecordSerializer<'s, 'w, W, S>;
+ type SerializeStructVariant = RecordSerializer<'s, 'w, W, S>;
+
+ fn serialize_bool(mut self, v: bool) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Boolean) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array([u8::from(v)])?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("bool", "Expected Schema::Boolean in variants"))?
+ }
+ }
+
+ fn serialize_i8(mut self, v: i8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i8", i32::from(v))
+ }
+
+ fn serialize_i16(mut self, v: i16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i16", i32::from(v))
+ }
+
+ fn serialize_i32(mut self, v: i32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("i32", v)
+ }
+
+ fn serialize_i64(mut self, v: i64) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("i64", v)
+ }
+
+ fn serialize_i128(mut self, v: i128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("i128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "i128",
+ r#"Expected Schema::Fixed(name: "i128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u8(mut self, v: u8) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u8", i32::from(v))
+ }
+
+ fn serialize_u16(mut self, v: u16) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_int("u16", i32::from(v))
+ }
+
+ fn serialize_u32(mut self, v: u32) -> Result<Self::Ok, Self::Error> {
+ self.checked_write_long("u32", i64::from(v))
+ }
+
+ fn serialize_u64(mut self, v: u64) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u64", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u64",
+ r#"Expected Schema::Fixed(name: "u64", size: 8) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_u128(mut self, v: u128) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema("u128", self.config.names)? {
+ Some((index, Schema::Fixed(FixedSchema { size: 16, .. }))) => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "u128",
+ r#"Expected Schema::Fixed(name: "u128", size: 16) in variants"#,
+ )),
+ }
+ }
+
+ fn serialize_f32(mut self, v: f32) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Float) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f32", "Expected Schema::Float in variants"))?
+ }
+ }
+
+ fn serialize_f64(mut self, v: f64) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Double) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_array(v.to_le_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("f64", "Expected Schema::Double in variants"))?
+ }
+ }
+
+ fn serialize_char(mut self, v: char) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.to_string().as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("char", "Expected Schema::String in variants"))?
+ }
+ }
+
+ fn serialize_str(mut self, v: &str) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::String) {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += self.write_bytes_with_len(v.as_bytes())?;
+ Ok(bytes_written)
+ } else {
+ Err(self.error("str", "Expected Schema::String in variants"))?
+ }
+ }
+
+ fn serialize_bytes(mut self, v: &[u8]) -> Result<Self::Ok, Self::Error> {
+ let (index, with_len) = match SER_BYTES_TYPE.get() {
+ BytesType::Bytes => {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Bytes) {
+ (index, true)
+ } else {
+ return Err(self.error("bytes", "Expected Schema::Bytes in variants"));
+ }
+ }
+ BytesType::Fixed => {
+ if let Some((index, _)) = self
+ .union
+ .find_fixed_of_size_n(v.len(), self.config.names)?
+ {
+ (index, false)
+ } else {
+ return Err(self.error(
+ "bytes",
+ format!("Expected Schema::Fixed(size: {}) in variants", v.len()),
+ ));
+ }
+ }
+ BytesType::Unset => {
+ let bytes_index = self.union.index_of_schema_kind(SchemaKind::Bytes);
+ let fixed_index = self
+ .union
+ .find_fixed_of_size_n(v.len(), self.config.names)?;
+ // Find the first variant that matches the bytes or fixed
+ match (bytes_index, fixed_index) {
+ (Some(bytes_index), Some((fixed_index, _))) => {
+ (bytes_index.min(fixed_index), bytes_index < fixed_index)
+ }
+ (Some(bytes_index), None) => (bytes_index, true),
+ (None, Some((fixed_index, _))) => (fixed_index, false),
+ (None, None) => {
+ return Err(self.error(
+ "bytes",
+ format!(
+ "Expected Schema::Bytes | Schema::Fixed(size: {}) in variants",
+ v.len()
+ ),
+ ));
+ }
+ }
+ }
+ };
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ if with_len {
+ bytes_written += self.write_bytes_with_len(v)?;
+ } else {
+ bytes_written += self.write_bytes(v)?;
+ }
+ Ok(bytes_written)
+ }
+
+ fn serialize_none(self) -> Result<Self::Ok, Self::Error> {
+ Err(self.error("none", "Nested unions are not supported"))
+ }
+
+ fn serialize_some<T>(self, _: &T) -> Result<Self::Ok, Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ Err(self.error("some", "Nested unions are not supported"))
+ }
+
+ fn serialize_unit(self) -> Result<Self::Ok, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Null) {
+ zig_i32(index as i32, &mut *self.writer)
+ } else {
+ Err(self.error("unit", "Expected Schema::Null in variants"))
+ }
+ }
+
+ fn serialize_unit_struct(self, name: &'static str) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Record(record))) if record.fields.is_empty() => {
+ zig_i32(index as i32, &mut *self.writer)
+ }
+ _ => Err(self.error(
+ "unit struct",
+ format!("Expected Schema::Record(name: {name}, fields: []) in variants"),
+ )),
+ }
+ }
+
+ fn serialize_unit_variant(
+ self,
+ name: &'static str,
+ variant_index: u32,
+ variant: &'static str,
+ ) -> Result<Self::Ok, Self::Error> {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Enum(schema))) if schema.symbols[variant_index as usize] == variant => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += zig_i32(variant_index as i32, &mut *self.writer)?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error("unit variant", format!("Expected Schema::Enum(name: {name}, symbols[{variant_index}] == {variant}) in variants"))),
+ }
+ }
+
+ fn serialize_newtype_struct<T>(
+ self,
+ name: &'static str,
+ value: &T,
+ ) -> Result<Self::Ok, Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Record(record))) if record.fields.len() == 1 => {
+ let mut bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ bytes_written += value.serialize(SchemaAwareSerializer::new(
+ self.writer,
+ &record.fields[0].schema,
+ self.config,
+ )?)?;
+ Ok(bytes_written)
+ }
+ _ => Err(self.error(
+ "newtype struct",
+ format!("Expected Schema::Record(name: {name}, fields: [_]) in variants"),
+ )),
+ }
+ }
+
+ fn serialize_newtype_variant<T>(
+ self,
+ _: &'static str,
+ _: u32,
+ _: &'static str,
+ _: &T,
+ ) -> Result<Self::Ok, Self::Error>
+ where
+ T: ?Sized + Serialize,
+ {
+ Err(self.error("newtype variant", "Nested unions are not supported"))
+ }
+
+ fn serialize_seq(self, len: Option<usize>) -> Result<Self::SerializeSeq, Self::Error> {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Array)
+ && let Schema::Array(array) = &self.union.variants()[index]
+ {
+ let bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ BlockSerializer::array(self.writer, array, self.config, len, Some(bytes_written))
+ } else {
+ Err(self.error("array", "Expected Schema::Array in variants"))
+ }
+ }
+
+ fn serialize_tuple(self, len: usize) -> Result<Self::SerializeTuple, Self::Error> {
+ if len == 0 {
+ if let Some(index) = self.union.index_of_schema_kind(SchemaKind::Null) {
+ let bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ Ok(TupleSerializer::unit(Some(bytes_written)))
+ } else {
+ Err(self.error("tuple", "Expected Schema::Null in variants for 0-tuple"))
+ }
+ } else if len == 1 {
+ Ok(TupleSerializer::one_union(
+ self.writer,
+ self.union,
+ self.config,
+ ))
+ } else if let Some((index, record)) = self
+ .union
+ .find_record_with_n_fields(len, self.config.names)?
+ {
+ let bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ Ok(TupleSerializer::many(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ } else {
+ Err(self.error(
+ "tuple",
+ format!(
+ "Expected Schema::Record(fields.len() == {len}) in variants for {len}-tuple"
+ ),
+ ))
+ }
+ }
+
+ fn serialize_tuple_struct(
+ self,
+ name: &'static str,
+ len: usize,
+ ) -> Result<Self::SerializeTupleStruct, Self::Error> {
+ match self.union.find_named_schema(name, self.config.names)? {
+ Some((index, Schema::Record(record))) if record.fields.len() == len => {
+ let bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ Ok(ManyTupleSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ }
+ _ => Err(self.error(
+ "tuple struct",
+ format!("Expected Schema::Record(name: {name}, fields.len() == {len}) in variants"),
+ )),
+ }
+ }
+
+ fn serialize_tuple_variant(
+ self,
+ _: &'static str,
+ _: u32,
+ _: &'static str,
+ _: usize,
+ ) -> Result<Self::SerializeTupleVariant, Self::Error> {
+ Err(self.error("tuple variant", "Nested unions are not supported"))
+ }
+
+ fn serialize_map(self, len: Option<usize>) -> Result<Self::SerializeMap, Self::Error> {
+ let map_index = self.union.index_of_schema_kind(SchemaKind::Map).map(|i| {
+ if let Schema::Map(map) = &self.union.variants()[i] {
+ (i, map)
+ } else {
+ unreachable!("SchemaKind is Map so Schema must also be a Map")
+ }
+ });
+ let record_index = if let Some(len) = len {
+ self.union
+ .find_record_with_n_fields(len, self.config.names)?
+ } else {
+ None
+ };
+ match (map_index, record_index) {
+ (Some((map_index, map)), Some((record_index, record))) => {
+ let bytes_written = zig_i32(map_index.min(record_index) as i32, &mut *self.writer)?;
+ if map_index < record_index {
+ MapOrRecordSerializer::map(
+ self.writer,
+ map,
+ self.config,
+ len,
+ Some(bytes_written),
+ )
+ } else {
+ Ok(MapOrRecordSerializer::record(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ }
+ }
+ (Some((map_index, map)), None) => {
+ let bytes_written = zig_i32(map_index as i32, &mut *self.writer)?;
+ MapOrRecordSerializer::map(self.writer, map, self.config, len, Some(bytes_written))
+ }
+ (None, Some((record_index, record))) => {
+ let bytes_written = zig_i32(record_index as i32, &mut *self.writer)?;
+ Ok(MapOrRecordSerializer::record(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ }
+ (None, None) => Err(self.error(
+ "map",
+ "Expected Schema::Map or Schema::Record for structs with flattened fields in variants",
+ )),
+ }
+ }
+
+ fn serialize_struct(
+ self,
+ name: &'static str,
+ _len: usize,
+ ) -> Result<Self::SerializeStruct, Self::Error> {
+ if let Some((index, Schema::Record(record))) =
+ self.union.find_named_schema(name, self.config.names)?
+ {
+ let bytes_written = zig_i32(index as i32, &mut *self.writer)?;
+ Ok(RecordSerializer::new(
+ self.writer,
+ record,
+ self.config,
+ Some(bytes_written),
+ ))
+ } else {
+ Err(self.error(
+ "struct",
+ format!("Expected Schema::Record(name: {name}) in variants"),
+ ))
+ }
+ }
+
+ fn serialize_struct_variant(
+ self,
+ _: &'static str,
+ _: u32,
+ _: &'static str,
+ _: usize,
+ ) -> Result<Self::SerializeStructVariant, Self::Error> {
+ Err(self.error("struct variant", "Nested unions are not supported"))
+ }
+
+ fn is_human_readable(&self) -> bool {
+ self.config.human_readable
+ }
+}
diff --git a/avro/src/writer/datum.rs b/avro/src/writer/datum.rs
index 539c894..3f79502 100644
--- a/avro/src/writer/datum.rs
+++ b/avro/src/writer/datum.rs
@@ -24,8 +24,9 @@ use crate::{
encode::encode_internal,
error::Details,
schema::{NamesRef, ResolvedSchema},
- serde::ser_schema::SchemaAwareWriteSerializer,
+ serde::ser_schema::{Config, SchemaAwareSerializer},
types::Value,
+ util::is_human_readable,
};
/// Writer for writing raw Avro data.
@@ -37,6 +38,8 @@ pub struct GenericDatumWriter<'s> {
schema: &'s Schema,
resolved: ResolvedSchema<'s>,
validate: bool,
+ human_readable: bool,
+ target_block_size: Option<usize>,
}
#[bon]
@@ -59,6 +62,19 @@ impl<'s> GenericDatumWriter<'s> {
/// written data unreadable.
#[builder(default = true)]
validate: bool,
+ /// At what block size to start a new block (for arrays and maps).
+ ///
+ /// This is a minimum value, the block size will always be larger than this except for the last
+ /// block.
+ ///
+ /// When set to `None` all values will be written in a single block. This can be faster as no
+ /// intermediate buffer is used, but seeking through written data will be slower.
+ target_block_size: Option<usize>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
+ ///
+ /// It is recommended to set this to `false`.
+ #[builder(default = is_human_readable())]
+ human_readable: bool,
) -> AvroResult<Self> {
let resolved = if let Some(resolved) = resolved_schemata {
resolved
@@ -69,6 +85,8 @@ impl<'s> GenericDatumWriter<'s> {
schema,
resolved,
validate,
+ human_readable,
+ target_block_size,
})
}
}
@@ -128,9 +146,12 @@ impl GenericDatumWriter<'_> {
writer: &mut W,
value: &T,
) -> AvroResult<usize> {
- let mut serializer =
- SchemaAwareWriteSerializer::new(writer, self.schema, self.resolved.get_names(), None);
- value.serialize(&mut serializer)
+ let config = Config {
+ names: self.resolved.get_names(),
+ target_block_size: self.target_block_size,
+ human_readable: self.human_readable,
+ };
+ value.serialize(SchemaAwareSerializer::new(writer, self.schema, config)?)
}
/// Serialize `T` to a [`Vec`].
@@ -181,8 +202,12 @@ pub fn write_avro_datum_ref<T: Serialize, W: Write>(
data: &T,
writer: &mut W,
) -> AvroResult<usize> {
- let mut serializer = SchemaAwareWriteSerializer::new(writer, schema, names, None);
- data.serialize(&mut serializer)
+ let config = Config {
+ names,
+ target_block_size: None,
+ human_readable: is_human_readable(),
+ };
+ data.serialize(SchemaAwareSerializer::new(writer, schema, config)?)
}
/// Deprecated. Use [`GenericDatumWriter`] instead.
diff --git a/avro/src/writer/mod.rs b/avro/src/writer/mod.rs
index 20f19a3..2d255a1 100644
--- a/avro/src/writer/mod.rs
+++ b/avro/src/writer/mod.rs
@@ -21,8 +21,9 @@ use crate::{
encode::{encode, encode_internal, encode_to_vec},
error::Details,
schema::{ResolvedSchema, Schema},
- serde::ser_schema::SchemaAwareWriteSerializer,
+ serde::ser_schema::{Config, SchemaAwareSerializer},
types::Value,
+ util::is_human_readable,
};
use serde::Serialize;
use std::{collections::HashMap, io::Write, mem::ManuallyDrop};
@@ -49,6 +50,8 @@ pub struct Writer<'a, W: Write> {
marker: [u8; 16],
has_header: bool,
user_metadata: HashMap<String, Value>,
+ human_readable: bool,
+ map_array_target_block_size: Option<usize>,
}
#[bon::bon]
@@ -67,6 +70,19 @@ impl<'a, W: Write> Writer<'a, W> {
#[builder(default = false)]
has_header: bool,
#[builder(default)] user_metadata: HashMap<String, Value>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
+ ///
+ /// It is recommended to set this to `false`.
+ #[builder(default = is_human_readable())]
+ human_readable: bool,
+ /// At what block size to start a new block (for arrays and maps).
+ ///
+ /// This is a minimum value, the block size will always be larger than this except for the last
+ /// block.
+ ///
+ /// When set to `None` all values will be written in a single block. This can be faster as no
+ /// intermediate buffer is used, but seeking through written data will be slower.
+ map_array_target_block_size: Option<usize>,
) -> AvroResult<Self> {
let resolved_schema = if let Some(schemata) = schemata {
ResolvedSchema::try_from(schemata)?
@@ -84,6 +100,8 @@ impl<'a, W: Write> Writer<'a, W> {
marker,
has_header,
user_metadata,
+ human_readable,
+ map_array_target_block_size,
})
}
}
@@ -271,13 +289,17 @@ impl<'a, W: Write> Writer<'a, W> {
pub fn append_ser<S: Serialize>(&mut self, value: S) -> AvroResult<usize> {
let n = self.maybe_write_header()?;
- let mut serializer = SchemaAwareWriteSerializer::new(
+ let config = Config {
+ names: self.resolved_schema.get_names(),
+ target_block_size: self.map_array_target_block_size,
+ human_readable: self.human_readable,
+ };
+
+ value.serialize(SchemaAwareSerializer::new(
&mut self.buffer,
self.schema,
- self.resolved_schema.get_names(),
- None,
- );
- value.serialize(&mut serializer)?;
+ config,
+ )?)?;
self.num_values += 1;
if self.buffer.len() >= self.block_size {
diff --git a/avro/src/writer/single_object.rs b/avro/src/writer/single_object.rs
index d5c2c8c..d34340c 100644
--- a/avro/src/writer/single_object.rs
+++ b/avro/src/writer/single_object.rs
@@ -17,10 +17,13 @@
use std::{io::Write, marker::PhantomData, ops::RangeInclusive};
+use bon::Builder;
use serde::Serialize;
+use crate::Error;
use crate::encode::encode_internal;
-use crate::serde::ser_schema::SchemaAwareWriteSerializer;
+use crate::serde::ser_schema::{Config, SchemaAwareSerializer};
+use crate::util::is_human_readable;
use crate::{
AvroResult, AvroSchema, Schema,
error::Details,
@@ -86,12 +89,35 @@ impl GenericSingleObjectWriter {
}
/// Writer that encodes messages according to the single object encoding v1 spec
+#[derive(Builder)]
pub struct SpecificSingleObjectWriter<T>
where
T: AvroSchema,
{
+ #[builder(
+ with = |schema: Schema| -> Result<_, Error> { ResolvedOwnedSchema::new(schema) },
+ default = ResolvedOwnedSchema::new(T::get_schema()).expect("AvroSchema implementation should create valid schemas")
+ )]
resolved: ResolvedOwnedSchema,
+ #[builder(
+ default = RabinFingerprintHeader::from_schema(resolved.get_root_schema()).build_header(),
+ with = |header_builder: impl HeaderBuilder| header_builder.build_header(),
+ )]
header: Vec<u8>,
+ /// Should [`Serialize`] implementations pick a human readable represenation.
+ ///
+ /// It is recommended to set this to `false`.
+ #[builder(default = is_human_readable())]
+ human_readable: bool,
+ /// At what block size to start a new block (for arrays and maps).
+ ///
+ /// This is a minimum value, the block size will always be larger than this except for the last
+ /// block.
+ ///
+ /// When set to `None` all values will be written in a single block. This can be faster as no
+ /// intermediate buffer is used, but seeking through written data will be slower.
+ target_block_size: Option<usize>,
+ #[builder(skip)]
_model: PhantomData<T>,
}
@@ -107,6 +133,8 @@ where
Ok(Self {
resolved,
header,
+ human_readable: is_human_readable(),
+ target_block_size: None,
_model: PhantomData,
})
}
@@ -117,6 +145,8 @@ where
Ok(Self {
resolved,
header,
+ human_readable: is_human_readable(),
+ target_block_size: None,
_model: PhantomData,
})
}
@@ -163,13 +193,17 @@ where
.write_all(&self.header)
.map_err(Details::WriteBytes)?;
- let mut serializer = SchemaAwareWriteSerializer::new(
+ let config = Config {
+ names: self.resolved.get_names(),
+ target_block_size: self.target_block_size,
+ human_readable: self.human_readable,
+ };
+
+ let bytes = data.serialize(SchemaAwareSerializer::new(
writer,
self.resolved.get_root_schema(),
- self.resolved.get_names(),
- None,
- );
- let bytes = data.serialize(&mut serializer)?;
+ config,
+ )?)?;
Ok(bytes + self.header.len())
}