You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@avro.apache.org by mg...@apache.org on 2022/04/16 20:52:20 UTC
[avro] branch master updated: AVRO-3479: [rust] Avro Schema Derive Proc Macro (#1631)
This is an automated email from the ASF dual-hosted git repository.
mgrigorov pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/avro.git
The following commit(s) were added to refs/heads/master by this push:
new cbc437202 AVRO-3479: [rust] Avro Schema Derive Proc Macro (#1631)
cbc437202 is described below
commit cbc4372029e7149c73db5a5aef491abdc5500b10
Author: Jack Klamer <jf...@gmail.com>
AuthorDate: Sat Apr 16 15:52:15 2022 -0500
AVRO-3479: [rust] Avro Schema Derive Proc Macro (#1631)
* port crate
* namespace port
* dev depends
* resolved against main
* Cons list tests
* rebased onto master resolution
* namespace attribute in derive
* std pointers
* References, testing, and refactoring
* [AVRO-3479] Clean up for PR
* AVRO-3479: Add missing ASL2 headers
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Minor improvements
Add TODOs
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* Schema assertions and PR comments
* test failure fixing
* add readme
* README + implementation guide + bug fix with enclosing namespaces
* AVRO-3479: Minor improvements
Fix typos.
Format the code/doc.
Apply suggestions by the IDE to use assert_eq!() instead of assert!()
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Fix typos
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Use darling crate to parse derive attributes
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* darling for NamedTypes and fields
* AVRO-3479 pr review naming
* AVRO-3479 doc comment doc and small tests
* AVRO-3479 featurize
* AVRO-3479 cargo engineering
* Fix a docu warning:
warning: unresolved link to `AvroSchemaComponent`
--> avro/src/schema.rs:1524:70
|
1524 | /// through `derive` feature. Do not implement directly, implement [`AvroSchemaComponent`]
| ^^^^^^^^^^^^^^^^^^^ no item named `AvroSchemaComponent` in scope
|
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
warning: `apache-avro` (lib doc) generated 1 warning
Finished dev [unoptimized + debuginfo] target(s) in 10.13s
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Rename avro_derive to apache-avro-derive
For consistency.
Add Cargo.toml metadata
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Use fqn for Mutex
For some reason Rustdoc build sometimes (not always!) complain that the
import of std::sync::Mutex is not used ...
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Update darling to 0.14.0
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Fix the version of apache-avro-derive
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Minor cleanups
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Inline a pub function that is used only in avro_derive
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Derive Schema::Long for u32
Validate successfully Value::Int into Schema::Long
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
* AVRO-3479: Bump dependencies to their latest versions
Signed-off-by: Martin Tzvetanov Grigorov <mg...@apache.org>
Co-authored-by: Martin Tzvetanov Grigorov <mg...@apache.org>
Co-authored-by: Martin Grigorov <ma...@users.noreply.github.com>
---
lang/rust/Cargo.toml | 1 +
lang/rust/avro/Cargo.toml | 6 +-
lang/rust/avro/src/schema.rs | 231 ++++++-
lang/rust/avro/src/types.rs | 1 +
lang/rust/{ => avro_derive}/Cargo.toml | 29 +-
lang/rust/avro_derive/README.md | 69 +++
lang/rust/avro_derive/src/lib.rs | 454 ++++++++++++++
lang/rust/avro_derive/tests/derive.rs | 1066 ++++++++++++++++++++++++++++++++
8 files changed, 1835 insertions(+), 22 deletions(-)
diff --git a/lang/rust/Cargo.toml b/lang/rust/Cargo.toml
index 9a8e9eaa2..9f188c065 100644
--- a/lang/rust/Cargo.toml
+++ b/lang/rust/Cargo.toml
@@ -18,4 +18,5 @@
[workspace]
members = [
"avro",
+ "avro_derive"
]
diff --git a/lang/rust/avro/Cargo.toml b/lang/rust/avro/Cargo.toml
index ffb3d69b8..be547acca 100644
--- a/lang/rust/avro/Cargo.toml
+++ b/lang/rust/avro/Cargo.toml
@@ -33,6 +33,7 @@ snappy = ["crc32fast", "snap"]
zstandard = ["zstd"]
bzip = ["bzip2"]
xz = ["xz2"]
+derive = ["apache-avro-derive" ]
[lib]
path = "src/lib.rs"
@@ -56,7 +57,7 @@ byteorder = "1.4.3"
bzip2 = { version = "0.4.3", optional = true }
crc32fast = { version = "1.3.2", optional = true }
digest = "0.10.3"
-libflate = "1.1.2"
+libflate = "1.2.0"
xz2 = { version = "0.1.6", optional = true }
num-bigint = "0.4.3"
rand = "0.8.5"
@@ -72,7 +73,8 @@ uuid = { version = "0.8.2", features = ["serde", "v4"] }
zerocopy = "0.6.1"
lazy_static = "1.4.0"
log = "0.4.16"
-zstd = { version = "0.11.0+zstd.1.5.2", optional = true }
+zstd = { version = "0.11.1+zstd.1.5.2", optional = true }
+apache-avro-derive = { version= "0.14.0", path = "../avro_derive", optional = true }
[dev-dependencies]
md-5 = "0.10.1"
diff --git a/lang/rust/avro/src/schema.rs b/lang/rust/avro/src/schema.rs
index a4ccb09a4..6134c8256 100644
--- a/lang/rust/avro/src/schema.rs
+++ b/lang/rust/avro/src/schema.rs
@@ -312,7 +312,7 @@ impl Name {
/// Name::new("some_namespace.some_name").unwrap()
/// );
/// ```
- pub(crate) fn fully_qualified_name(&self, enclosing_namespace: &Namespace) -> Name {
+ pub fn fully_qualified_name(&self, enclosing_namespace: &Namespace) -> Name {
Name {
name: self.name.clone(),
namespace: self
@@ -1006,7 +1006,8 @@ impl Parser {
schema: &Schema,
aliases: &Aliases,
) {
- // FIXME, this should be globally aware, so if there is something overwriting something else then there is an ambiguois schema definition. An apropriate error should be thrown
+ // FIXME, this should be globally aware, so if there is something overwriting something
+ // else then there is an ambiguous schema definition. An appropriate error should be thrown
self.parsed_schemas
.insert(fully_qualified_name.clone(), schema.clone());
self.resolving_schemas.remove(fully_qualified_name);
@@ -1526,6 +1527,204 @@ fn field_ordering_position(field: &str) -> Option<usize> {
.map(|pos| pos + 1)
}
+/// Trait for types that serve as an Avro data model. Derive implementation available
+/// through `derive` feature. Do not implement directly, implement [`derive::AvroSchemaComponent`]
+/// to get this trait through a blanket implementation.
+pub trait AvroSchema {
+ fn get_schema() -> Schema;
+}
+
+#[cfg(feature = "derive")]
+pub mod derive {
+ use super::*;
+
+ /// Trait for types that serve as fully defined components inside an Avro data model. Derive
+ /// implementation available through `derive` feature. This is what is implemented by
+ /// the `derive(AvroSchema)` macro.
+ ///
+ /// # Implementation guide
+ ///
+ ///### Simple implementation
+ /// To construct a non named simple schema, it is possible to ignore the input argument making the
+ /// general form implementation look like
+ /// ```ignore
+ /// impl AvroSchemaComponent for AType {
+ /// fn get_schema_in_ctxt(_: &mut Names, _: &Namespace) -> Schema {
+ /// Schema::?
+ /// }
+ ///}
+ /// ```
+ /// ### Passthrough implementation
+ /// To construct a schema for a Type that acts as in "inner" type, such as for smart pointers, simply
+ /// pass through the arguments to the inner type
+ /// ```ignore
+ /// impl AvroSchemaComponent for PassthroughType {
+ /// fn get_schema_in_ctxt(named_schemas: &mut Names, enclosing_namespace: &Namespace) -> Schema {
+ /// InnerType::get_schema_in_ctxt(names, enclosing_namespace)
+ /// }
+ ///}
+ /// ```
+ ///### Complex implementation
+ /// To implement this for Named schema there is a general form needed to avoid creating invalid
+ /// schemas or infinite loops.
+ /// ```ignore
+ /// impl AvroSchemaComponent for ComplexType {
+ /// fn get_schema_in_ctxt(named_schemas: &mut Names, enclosing_namespace: &Namespace) -> Schema {
+ /// // Create the fully qualified name for your type given the enclosing namespace
+ /// let name = apache_avro::schema::Name::new("MyName")
+ /// .expect("Unable to parse schema name")
+ /// .fully_qualified_name(enclosing_namespace);
+ /// let enclosing_namespace = &name.namespace;
+ /// // Check, if your name is already defined, and if so, return a ref to that name
+ /// if named_schemas.contains_key(&name) {
+ /// apache_avro::schema::Schema::Ref{name: name.clone()}
+ /// } else {
+ /// named_schemas.insert(name.clone(), apache_avro::schema::Schema::Ref{name: name.clone()});
+ /// // YOUR SCHEMA DEFINITION HERE with the name equivalent to "MyName".
+ /// // For non-simple sub types delegate to their implementation of AvroSchemaComponent
+ /// }
+ /// }
+ ///}
+ /// ```
+ pub trait AvroSchemaComponent {
+ fn get_schema_in_ctxt(named_schemas: &mut Names, enclosing_namespace: &Namespace)
+ -> Schema;
+ }
+
+ impl<T> AvroSchema for T
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema() -> Schema {
+ T::get_schema_in_ctxt(&mut HashMap::default(), &Option::None)
+ }
+ }
+
+ macro_rules! impl_schema(
+ ($type:ty, $variant_constructor:expr) => (
+ impl AvroSchemaComponent for $type {
+ fn get_schema_in_ctxt(_: &mut Names, _: &Namespace) -> Schema {
+ $variant_constructor
+ }
+ }
+ );
+ );
+
+ impl_schema!(i8, Schema::Int);
+ impl_schema!(i16, Schema::Int);
+ impl_schema!(i32, Schema::Int);
+ impl_schema!(i64, Schema::Long);
+ impl_schema!(u8, Schema::Int);
+ impl_schema!(u16, Schema::Int);
+ impl_schema!(u32, Schema::Long);
+ impl_schema!(f32, Schema::Float);
+ impl_schema!(f64, Schema::Double);
+ impl_schema!(String, Schema::String);
+ impl_schema!(uuid::Uuid, Schema::Uuid);
+ impl_schema!(core::time::Duration, Schema::Duration);
+
+ impl<T> AvroSchemaComponent for Vec<T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ Schema::Array(Box::new(T::get_schema_in_ctxt(
+ named_schemas,
+ enclosing_namespace,
+ )))
+ }
+ }
+
+ impl<T> AvroSchemaComponent for Option<T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ let inner_schema = T::get_schema_in_ctxt(named_schemas, enclosing_namespace);
+ Schema::Union(UnionSchema {
+ schemas: vec![Schema::Null, inner_schema.clone()],
+ variant_index: vec![Schema::Null, inner_schema]
+ .iter()
+ .enumerate()
+ .map(|(idx, s)| (SchemaKind::from(s), idx))
+ .collect(),
+ })
+ }
+ }
+
+ impl<T> AvroSchemaComponent for Map<String, T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ Schema::Map(Box::new(T::get_schema_in_ctxt(
+ named_schemas,
+ enclosing_namespace,
+ )))
+ }
+ }
+
+ impl<T> AvroSchemaComponent for HashMap<String, T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ Schema::Map(Box::new(T::get_schema_in_ctxt(
+ named_schemas,
+ enclosing_namespace,
+ )))
+ }
+ }
+
+ impl<T> AvroSchemaComponent for Box<T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ T::get_schema_in_ctxt(named_schemas, enclosing_namespace)
+ }
+ }
+
+ impl<T> AvroSchemaComponent for std::sync::Mutex<T>
+ where
+ T: AvroSchemaComponent,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ T::get_schema_in_ctxt(named_schemas, enclosing_namespace)
+ }
+ }
+
+ impl<T> AvroSchemaComponent for Cow<'_, T>
+ where
+ T: AvroSchemaComponent + Clone,
+ {
+ fn get_schema_in_ctxt(
+ named_schemas: &mut Names,
+ enclosing_namespace: &Namespace,
+ ) -> Schema {
+ T::get_schema_in_ctxt(named_schemas, enclosing_namespace)
+ }
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -2589,7 +2788,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_record_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2628,7 +2827,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_record_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2662,7 +2861,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_enum_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2696,7 +2895,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_enum_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2730,7 +2929,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_fixed_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2764,7 +2963,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.inner_fixed_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2804,7 +3003,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "inner_space.inner_record_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2839,7 +3038,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "inner_space.inner_enum_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2874,7 +3073,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "inner_space.inner_fixed_name"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -2925,7 +3124,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 3);
+ assert_eq!(rs.get_names().len(), 3);
for s in &[
"space.record_name",
"space.middle_record_name",
@@ -2981,7 +3180,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 3);
+ assert_eq!(rs.get_names().len(), 3);
for s in &[
"space.record_name",
"middle_namespace.middle_record_name",
@@ -3038,7 +3237,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 3);
+ assert_eq!(rs.get_names().len(), 3);
for s in &[
"space.record_name",
"middle_namespace.middle_record_name",
@@ -3081,7 +3280,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.in_array_record"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
@@ -3120,7 +3319,7 @@ mod tests {
"#;
let schema = Schema::parse_str(schema).unwrap();
let rs = ResolvedSchema::try_from(&schema).expect("Schema didn't successfully parse");
- assert!(rs.get_names().len() == 2);
+ assert_eq!(rs.get_names().len(), 2);
for s in &["space.record_name", "space.in_map_record"] {
assert!(rs.get_names().contains_key(&Name::new(s).unwrap()));
}
diff --git a/lang/rust/avro/src/types.rs b/lang/rust/avro/src/types.rs
index d77ded887..25d968120 100644
--- a/lang/rust/avro/src/types.rs
+++ b/lang/rust/avro/src/types.rs
@@ -372,6 +372,7 @@ impl Value {
(&Value::Int(_), &Schema::Int) => None,
(&Value::Int(_), &Schema::Date) => None,
(&Value::Int(_), &Schema::TimeMillis) => None,
+ (&Value::Int(_), &Schema::Long) => None,
(&Value::Long(_), &Schema::Long) => None,
(&Value::Long(_), &Schema::TimeMicros) => None,
(&Value::Long(_), &Schema::TimestampMillis) => None,
diff --git a/lang/rust/Cargo.toml b/lang/rust/avro_derive/Cargo.toml
similarity index 51%
copy from lang/rust/Cargo.toml
copy to lang/rust/avro_derive/Cargo.toml
index 9a8e9eaa2..e16e9ea95 100644
--- a/lang/rust/Cargo.toml
+++ b/lang/rust/avro_derive/Cargo.toml
@@ -15,7 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-[workspace]
-members = [
- "avro",
-]
+[package]
+name = "apache-avro-derive"
+version = "0.14.0"
+authors = ["Apache Avro team <de...@avro.apache.org>"]
+description = "A library for deriving Avro schemata from Rust structs and enums"
+license = "Apache-2.0"
+readme = "README.md"
+repository = "https://github.com/apache/avro"
+edition = "2018"
+keywords = ["avro", "data", "serialization", "derive"]
+categories = ["encoding"]
+documentation = "https://docs.rs/apache-avro-derive"
+
+[lib]
+proc-macro = true
+
+[dependencies]
+syn = {version= "1.0.91", features=["full", "fold"]}
+quote = "1.0.18"
+proc-macro2 = "1.0.37"
+darling = "0.14.0"
+
+[dev-dependencies]
+serde = { version = "1.0.136", features = ["derive"] }
+apache-avro = { version = "0.14.0", path = "../avro", features = ["derive"] }
diff --git a/lang/rust/avro_derive/README.md b/lang/rust/avro_derive/README.md
new file mode 100644
index 000000000..6faa215f5
--- /dev/null
+++ b/lang/rust/avro_derive/README.md
@@ -0,0 +1,69 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+
+# avro_derive
+
+A proc-macro module for automatically deriving the avro schema for structs or enums. The macro produces the logic necessary to implement the `AvroSchema` trait for the type.
+
+```rust
+pub trait AvroSchema {
+ // constructs the schema for the type
+ fn get_schema() -> Schema;
+}
+```
+## How-to use
+Add the "derive" feature to your apache-avro dependency inside cargo.toml
+```
+apache-avro = { version = "X.Y.Z", features = ["derive"] }
+```
+
+Add to your data model
+```rust
+#[derive(AvroSchema)]
+struct Test {
+ a: i64,
+ b: String,
+}
+```
+
+
+### Example
+```rust
+use apache_avro::Writer;
+
+#[derive(Debug, Serialize, AvroSchema)]
+struct Test {
+ a: i64,
+ b: String,
+}
+// derived schema, always valid or code fails to compile with a descriptive message
+let schema = Test::get_schema();
+
+let mut writer = Writer::new(&schema, Vec::new());
+let test = Test {
+ a: 27,
+ b: "foo".to_owned(),
+};
+writer.append_ser(test).unwrap();
+let encoded = writer.into_inner();
+```
+
+### Compatibility Notes
+This module is designed to work in concert with the Serde implemenation. If your use case dictates needing to manually convert to a `Value` type in order to encode then the derived schema may not be correct.
diff --git a/lang/rust/avro_derive/src/lib.rs b/lang/rust/avro_derive/src/lib.rs
new file mode 100644
index 000000000..96575e090
--- /dev/null
+++ b/lang/rust/avro_derive/src/lib.rs
@@ -0,0 +1,454 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+extern crate darling;
+
+use darling::FromAttributes;
+use proc_macro2::{Span, TokenStream};
+use quote::quote;
+
+use syn::{
+ parse_macro_input, spanned::Spanned, AttrStyle, Attribute, DeriveInput, Error, Type, TypePath,
+};
+
+#[derive(FromAttributes)]
+#[darling(attributes(avro))]
+struct FieldOptions {
+ #[darling(default)]
+ doc: Option<String>,
+}
+
+#[derive(FromAttributes)]
+#[darling(attributes(avro))]
+struct NamedTypeOptions {
+ #[darling(default)]
+ namespace: Option<String>,
+ #[darling(default)]
+ doc: Option<String>,
+}
+
+#[proc_macro_derive(AvroSchema, attributes(avro))]
+// Templated from Serde
+pub fn proc_macro_derive_avro_schema(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
+ let mut input = parse_macro_input!(input as DeriveInput);
+ derive_avro_schema(&mut input)
+ .unwrap_or_else(to_compile_errors)
+ .into()
+}
+
+fn derive_avro_schema(input: &mut DeriveInput) -> Result<TokenStream, Vec<syn::Error>> {
+ let named_type_options =
+ NamedTypeOptions::from_attributes(&input.attrs[..]).map_err(darling_to_syn)?;
+ let full_schema_name = vec![named_type_options.namespace, Some(input.ident.to_string())]
+ .into_iter()
+ .flatten()
+ .collect::<Vec<String>>()
+ .join(".");
+ let schema_def = match &input.data {
+ syn::Data::Struct(s) => get_data_struct_schema_def(
+ &full_schema_name,
+ named_type_options
+ .doc
+ .or_else(|| extract_outer_doc(&input.attrs)),
+ s,
+ input.ident.span(),
+ )?,
+ syn::Data::Enum(e) => get_data_enum_schema_def(
+ &full_schema_name,
+ named_type_options
+ .doc
+ .or_else(|| extract_outer_doc(&input.attrs)),
+ e,
+ input.ident.span(),
+ )?,
+ _ => {
+ return Err(vec![Error::new(
+ input.ident.span(),
+ "AvroSchema derive only works for structs and simple enums ",
+ )])
+ }
+ };
+
+ let ident = &input.ident;
+ let (impl_generics, ty_generics, where_clause) = input.generics.split_for_impl();
+ Ok(quote! {
+ impl #impl_generics apache_avro::schema::derive::AvroSchemaComponent for #ident #ty_generics #where_clause {
+ fn get_schema_in_ctxt(named_schemas: &mut HashMap<apache_avro::schema::Name, apache_avro::schema::Schema>, enclosing_namespace: &Option<String>) -> apache_avro::schema::Schema {
+ let name = apache_avro::schema::Name::new(#full_schema_name).expect(&format!("Unable to parse schema name {}", #full_schema_name)[..]).fully_qualified_name(enclosing_namespace);
+ let enclosing_namespace = &name.namespace;
+ if named_schemas.contains_key(&name) {
+ apache_avro::schema::Schema::Ref{name: name.clone()}
+ } else {
+ named_schemas.insert(name.clone(), apache_avro::schema::Schema::Ref{name: name.clone()});
+ #schema_def
+ }
+ }
+ }
+ })
+}
+
+fn get_data_struct_schema_def(
+ full_schema_name: &str,
+ record_doc: Option<String>,
+ s: &syn::DataStruct,
+ error_span: Span,
+) -> Result<TokenStream, Vec<syn::Error>> {
+ let mut record_field_exprs = vec![];
+ match s.fields {
+ syn::Fields::Named(ref a) => {
+ for (position, field) in a.named.iter().enumerate() {
+ let name = field.ident.as_ref().unwrap().to_string(); // we know everything has a name
+ let field_documented =
+ FieldOptions::from_attributes(&field.attrs[..]).map_err(darling_to_syn)?;
+ let doc = preserve_optional(field_documented.doc);
+ let schema_expr = type_to_schema_expr(&field.ty)?;
+ let position = position;
+ record_field_exprs.push(quote! {
+ apache_avro::schema::RecordField {
+ name: #name.to_string(),
+ doc: #doc,
+ default: Option::None,
+ schema: #schema_expr,
+ order: apache_avro::schema::RecordFieldOrder::Ascending,
+ position: #position,
+ }
+ });
+ }
+ }
+ syn::Fields::Unnamed(_) => {
+ return Err(vec![Error::new(
+ error_span,
+ "AvroSchema derive does not work for tuple structs",
+ )])
+ }
+ syn::Fields::Unit => {
+ return Err(vec![Error::new(
+ error_span,
+ "AvroSchema derive does not work for unit structs",
+ )])
+ }
+ }
+ let record_doc = preserve_optional(record_doc);
+ Ok(quote! {
+ let schema_fields = vec![#(#record_field_exprs),*];
+ let name = apache_avro::schema::Name::new(#full_schema_name).expect(&format!("Unable to parse struct name for schema {}", #full_schema_name)[..]);
+ let lookup: HashMap<String, usize> = schema_fields
+ .iter()
+ .map(|field| (field.name.to_owned(), field.position))
+ .collect();
+ apache_avro::schema::Schema::Record {
+ name,
+ aliases: None,
+ doc: #record_doc,
+ fields: schema_fields,
+ lookup,
+ }
+ })
+}
+
+fn get_data_enum_schema_def(
+ full_schema_name: &str,
+ doc: Option<String>,
+ e: &syn::DataEnum,
+ error_span: Span,
+) -> Result<TokenStream, Vec<syn::Error>> {
+ let doc = preserve_optional(doc);
+ if e.variants.iter().all(|v| syn::Fields::Unit == v.fields) {
+ let symbols: Vec<String> = e
+ .variants
+ .iter()
+ .map(|variant| variant.ident.to_string())
+ .collect();
+ Ok(quote! {
+ apache_avro::schema::Schema::Enum {
+ name: apache_avro::schema::Name::new(#full_schema_name).expect(&format!("Unable to parse enum name for schema {}", #full_schema_name)[..]),
+ aliases: None,
+ doc: #doc,
+ symbols: vec![#(#symbols.to_owned()),*]
+ }
+ })
+ } else {
+ Err(vec![Error::new(
+ error_span,
+ "AvroSchema derive does not work for enums with non unit structs",
+ )])
+ }
+}
+
+/// Takes in the Tokens of a type and returns the tokens of an expression with return type `Schema`
+fn type_to_schema_expr(ty: &Type) -> Result<TokenStream, Vec<syn::Error>> {
+ if let Type::Path(p) = ty {
+ let type_string = p.path.segments.last().unwrap().ident.to_string();
+
+ let schema = match &type_string[..] {
+ "bool" => quote! {Schema::Boolean},
+ "i8" | "i16" | "i32" | "u8" | "u16" => quote! {apache_avro::schema::Schema::Int},
+ "i64" => quote! {apache_avro::schema::Schema::Long},
+ "f32" => quote! {apache_avro::schema::Schema::Float},
+ "f64" => quote! {apache_avro::schema::Schema::Double},
+ "String" | "str" => quote! {apache_avro::schema::Schema::String},
+ "char" => {
+ return Err(vec![syn::Error::new_spanned(
+ ty,
+ "AvroSchema: Cannot guarantee successful deserialization of this type",
+ )])
+ }
+ "u64" => {
+ return Err(vec![syn::Error::new_spanned(
+ ty,
+ "Cannot guarantee successful serialization of this type due to overflow concerns",
+ )])
+ } // Can't guarantee serialization type
+ _ => {
+ // Fails when the type does not implement AvroSchemaComponent directly
+ // TODO check and error report with something like https://docs.rs/quote/1.0.15/quote/macro.quote_spanned.html#example
+ type_path_schema_expr(p)
+ }
+ };
+ Ok(schema)
+ } else if let Type::Array(ta) = ty {
+ let inner_schema_expr = type_to_schema_expr(&ta.elem)?;
+ Ok(quote! {apache_avro::schema::Schema::Array(Box::new(#inner_schema_expr))})
+ } else if let Type::Reference(tr) = ty {
+ type_to_schema_expr(&tr.elem)
+ } else {
+ Err(vec![syn::Error::new_spanned(
+ ty,
+ format!("Unable to generate schema for type: {:?}", ty),
+ )])
+ }
+}
+
+/// Generates the schema def expression for fully qualified type paths using the associated function
+/// - `A -> <A as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt()`
+/// - `A<T> -> <A<T> as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt()`
+fn type_path_schema_expr(p: &TypePath) -> TokenStream {
+ quote! {<#p as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt(named_schemas, enclosing_namespace)}
+}
+
+/// Stolen from serde
+fn to_compile_errors(errors: Vec<syn::Error>) -> proc_macro2::TokenStream {
+ let compile_errors = errors.iter().map(syn::Error::to_compile_error);
+ quote!(#(#compile_errors)*)
+}
+
+fn extract_outer_doc(attributes: &[Attribute]) -> Option<String> {
+ let doc = attributes
+ .iter()
+ .filter(|attr| attr.style == AttrStyle::Outer && attr.path.is_ident("doc"))
+ .map(|attr| {
+ let mut tokens = attr.tokens.clone().into_iter();
+ tokens.next(); // skip the Punct
+ let to_trim: &[char] = &['"', ' '];
+ tokens
+ .next() // use the Literal
+ .unwrap()
+ .to_string()
+ .trim_matches(to_trim)
+ .to_string()
+ })
+ .collect::<Vec<String>>()
+ .join("\n");
+ if doc.is_empty() {
+ None
+ } else {
+ Some(doc)
+ }
+}
+
+fn preserve_optional(op: Option<impl quote::ToTokens>) -> TokenStream {
+ match op {
+ Some(tt) => quote! {Some(#tt.into())},
+ None => quote! {None},
+ }
+}
+
+fn darling_to_syn(e: darling::Error) -> Vec<syn::Error> {
+ let msg = format!("{}", e);
+ let token_errors = e.write_errors();
+ vec![syn::Error::new(token_errors.span(), msg)]
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ #[test]
+ fn basic_case() {
+ let test_struct = quote! {
+ struct A {
+ a: i32,
+ b: String
+ }
+ };
+
+ match syn::parse2::<DeriveInput>(test_struct) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_ok())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn tuple_struct_unsupported() {
+ let test_tuple_struct = quote! {
+ struct B (i32, String);
+ };
+
+ match syn::parse2::<DeriveInput>(test_tuple_struct) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_err())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn unit_struct_unsupported() {
+ let test_tuple_struct = quote! {
+ struct AbsoluteUnit;
+ };
+
+ match syn::parse2::<DeriveInput>(test_tuple_struct) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_err())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn struct_with_optional() {
+ let struct_with_optional = quote! {
+ struct Test4 {
+ a : Option<i32>
+ }
+ };
+ match syn::parse2::<DeriveInput>(struct_with_optional) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_ok())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn test_basic_enum() {
+ let basic_enum = quote! {
+ enum Basic {
+ A,
+ B,
+ C,
+ D
+ }
+ };
+ match syn::parse2::<DeriveInput>(basic_enum) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_ok())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn test_non_basic_enum() {
+ let non_basic_enum = quote! {
+ enum Basic {
+ A(i32),
+ B,
+ C,
+ D
+ }
+ };
+ match syn::parse2::<DeriveInput>(non_basic_enum) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_err())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn test_namespace() {
+ let test_struct = quote! {
+ #[avro(namespace = "namespace.testing")]
+ struct A {
+ a: i32,
+ b: String
+ }
+ };
+
+ match syn::parse2::<DeriveInput>(test_struct) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_ok());
+ assert!(derive_avro_schema(&mut input)
+ .unwrap()
+ .to_string()
+ .contains("namespace.testing"))
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn test_reference() {
+ let test_reference_struct = quote! {
+ struct A<'a> {
+ a: &'a Vec<i32>,
+ b: &'static str
+ }
+ };
+
+ match syn::parse2::<DeriveInput>(test_reference_struct) {
+ Ok(mut input) => {
+ assert!(derive_avro_schema(&mut input).is_ok())
+ }
+ Err(error) => panic!(
+ "Failed to parse as derive input when it should be able to. Error: {:?}",
+ error
+ ),
+ };
+ }
+
+ #[test]
+ fn test_trait_cast() {
+ assert_eq!(type_path_schema_expr(&syn::parse2::<TypePath>(quote!{i32}).unwrap()).to_string(), quote!{<i32 as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt(named_schemas, enclosing_namespace)}.to_string());
+ assert_eq!(type_path_schema_expr(&syn::parse2::<TypePath>(quote!{Vec<T>}).unwrap()).to_string(), quote!{<Vec<T> as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt(named_schemas, enclosing_namespace)}.to_string());
+ assert_eq!(type_path_schema_expr(&syn::parse2::<TypePath>(quote!{AnyType}).unwrap()).to_string(), quote!{<AnyType as apache_avro::schema::derive::AvroSchemaComponent>::get_schema_in_ctxt(named_schemas, enclosing_namespace)}.to_string());
+ }
+}
diff --git a/lang/rust/avro_derive/tests/derive.rs b/lang/rust/avro_derive/tests/derive.rs
new file mode 100644
index 000000000..8ac95755f
--- /dev/null
+++ b/lang/rust/avro_derive/tests/derive.rs
@@ -0,0 +1,1066 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use apache_avro::{
+ from_value,
+ schema::{derive::AvroSchemaComponent, AvroSchema},
+ Reader, Schema, Writer,
+};
+use apache_avro_derive::*;
+use serde::{de::DeserializeOwned, ser::Serialize};
+use std::collections::HashMap;
+
+#[macro_use]
+extern crate serde;
+
+#[cfg(test)]
+mod test_derive {
+ use std::{
+ borrow::{Borrow, Cow},
+ sync::Mutex,
+ };
+
+ use super::*;
+
+ /// Takes in a type that implements the right combination of traits and runs it through a Serde Cycle and asserts the result is the same
+ fn serde_assert<T>(obj: T)
+ where
+ T: std::fmt::Debug + Serialize + DeserializeOwned + AvroSchema + Clone + PartialEq,
+ {
+ assert_eq!(obj, serde(obj.clone()));
+ }
+
+ fn serde<T>(obj: T) -> T
+ where
+ T: Serialize + DeserializeOwned + AvroSchema,
+ {
+ de(ser(obj))
+ }
+
+ fn ser<T>(obj: T) -> Vec<u8>
+ where
+ T: Serialize + AvroSchema,
+ {
+ let schema = T::get_schema();
+ let mut writer = Writer::new(&schema, Vec::new());
+ if let Err(e) = writer.append_ser(obj) {
+ panic!("{:?}", e);
+ }
+ writer.into_inner().unwrap()
+ }
+
+ fn de<T>(encoded: Vec<u8>) -> T
+ where
+ T: DeserializeOwned + AvroSchema,
+ {
+ assert!(!encoded.is_empty());
+ let schema = T::get_schema();
+ let reader = Reader::with_schema(&schema, &encoded[..]).unwrap();
+ for res in reader {
+ match res {
+ Ok(value) => {
+ return from_value::<T>(&value).unwrap();
+ }
+ Err(e) => panic!("{:?}", e),
+ }
+ }
+ unreachable!()
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestBasic {
+ a: i32,
+ b: String,
+ }
+
+ #[test]
+ fn test_smoke_test() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestBasic",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int"
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestBasic::get_schema());
+ let test = TestBasic {
+ a: 27,
+ b: "foo".to_owned(),
+ };
+ serde_assert(test);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ #[avro(namespace = "com.testing.namespace")]
+ struct TestBasicNamespace {
+ a: i32,
+ b: String,
+ }
+
+ #[test]
+ fn test_basic_namespace() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"com.testing.namespace.TestBasicNamespace",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int"
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestBasicNamespace::get_schema());
+ if let Schema::Record { name, .. } = TestBasicNamespace::get_schema() {
+ assert_eq!("com.testing.namespace".to_owned(), name.namespace.unwrap())
+ } else {
+ panic!("TestBasicNamespace schema must be a record schema")
+ }
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ #[avro(namespace = "com.testing.complex.namespace")]
+ struct TestComplexNamespace {
+ a: TestBasicNamespace,
+ b: String,
+ }
+
+ #[test]
+ fn test_complex_namespace() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"com.testing.complex.namespace.TestComplexNamespace",
+ "fields":[
+ {
+ "name":"a",
+ "type":{
+ "type":"record",
+ "name":"com.testing.namespace.TestBasicNamespace",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int"
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestComplexNamespace::get_schema());
+ if let Schema::Record { name, fields, .. } = TestComplexNamespace::get_schema() {
+ assert_eq!(
+ "com.testing.complex.namespace".to_owned(),
+ name.namespace.unwrap()
+ );
+ let inner_schema = fields
+ .iter()
+ .filter(|field| field.name == "a")
+ .map(|field| &field.schema)
+ .next();
+ if let Some(Schema::Record { name, .. }) = inner_schema {
+ assert_eq!(
+ "com.testing.namespace".to_owned(),
+ name.namespace.clone().unwrap()
+ )
+ } else {
+ panic!("Field 'a' must have a record schema")
+ }
+ } else {
+ panic!("TestComplexNamespace schema must be a record schema")
+ }
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestAllSupportedBaseTypes {
+ //Basics test
+ a: bool,
+ b: i8,
+ c: i16,
+ d: i32,
+ e: u8,
+ f: u16,
+ g: i64,
+ h: f32,
+ i: f64,
+ j: String,
+ }
+
+ #[test]
+ fn test_basic_types() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestAllSupportedBaseTypes",
+ "fields":[
+ {
+ "name":"a",
+ "type": "boolean"
+ },
+ {
+ "name":"b",
+ "type":"int"
+ },
+ {
+ "name":"c",
+ "type":"int"
+ },
+ {
+ "name":"d",
+ "type":"int"
+ },
+ {
+ "name":"e",
+ "type":"int"
+ },
+ {
+ "name":"f",
+ "type":"int"
+ },
+ {
+ "name":"g",
+ "type":"long"
+ },
+ {
+ "name":"h",
+ "type":"float"
+ },
+ {
+ "name":"i",
+ "type":"double"
+ },
+ {
+ "name":"j",
+ "type":"string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestAllSupportedBaseTypes::get_schema());
+ // TODO mgrigorov Use property based testing in the future
+ let all_basic = TestAllSupportedBaseTypes {
+ a: true,
+ b: 8_i8,
+ c: 16_i16,
+ d: 32_i32,
+ e: 8_u8,
+ f: 16_u16,
+ g: 64_i64,
+ h: 32.3333_f32,
+ i: 64.4444_f64,
+ j: "testing string".to_owned(),
+ };
+ serde_assert(all_basic);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestNested {
+ a: i32,
+ b: TestAllSupportedBaseTypes,
+ }
+
+ #[test]
+ fn test_inner_struct() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestNested",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int"
+ },
+ {
+ "name":"b",
+ "type":{
+ "type":"record",
+ "name":"TestAllSupportedBaseTypes",
+ "fields":[
+ {
+ "name":"a",
+ "type": "boolean"
+ },
+ {
+ "name":"b",
+ "type":"int"
+ },
+ {
+ "name":"c",
+ "type":"int"
+ },
+ {
+ "name":"d",
+ "type":"int"
+ },
+ {
+ "name":"e",
+ "type":"int"
+ },
+ {
+ "name":"f",
+ "type":"int"
+ },
+ {
+ "name":"g",
+ "type":"long"
+ },
+ {
+ "name":"h",
+ "type":"float"
+ },
+ {
+ "name":"i",
+ "type":"double"
+ },
+ {
+ "name":"j",
+ "type":"string"
+ }
+ ]
+ }
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestNested::get_schema());
+ // TODO mgrigorov Use property based testing in the future
+ let all_basic = TestAllSupportedBaseTypes {
+ a: true,
+ b: 8_i8,
+ c: 16_i16,
+ d: 32_i32,
+ e: 8_u8,
+ f: 16_u16,
+ g: 64_i64,
+ h: 32.3333_f32,
+ i: 64.4444_f64,
+ j: "testing string".to_owned(),
+ };
+ let inner_struct = TestNested {
+ a: -1600,
+ b: all_basic,
+ };
+ serde_assert(inner_struct);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestOptional {
+ a: Option<i32>,
+ }
+
+ #[test]
+ fn test_optional_field_some() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestOptional",
+ "fields":[
+ {
+ "name":"a",
+ "type":["null","int"]
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestOptional::get_schema());
+ let optional_field = TestOptional { a: Some(4) };
+ serde_assert(optional_field);
+ }
+
+ #[test]
+ fn test_optional_field_none() {
+ let optional_field = TestOptional { a: None };
+ serde_assert(optional_field);
+ }
+
+ /// Generic Containers
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestGeneric<T: AvroSchemaComponent> {
+ a: String,
+ b: Vec<T>,
+ c: HashMap<String, T>,
+ }
+
+ #[test]
+ fn test_generic_container_1() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestGeneric",
+ "fields":[
+ {
+ "name":"a",
+ "type":"string"
+ },
+ {
+ "name":"b",
+ "type": {
+ "type":"array",
+ "items":"int"
+ }
+ },
+ {
+ "name":"c",
+ "type": {
+ "type":"map",
+ "values":"int"
+ }
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestGeneric::<i32>::get_schema());
+ let test_generic = TestGeneric::<i32> {
+ a: "testing".to_owned(),
+ b: vec![0, 1, 2, 3],
+ c: vec![("key".to_owned(), 3)].into_iter().collect(),
+ };
+ serde_assert(test_generic);
+ }
+
+ #[test]
+ fn test_generic_container_2() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestGeneric",
+ "fields":[
+ {
+ "name":"a",
+ "type":"string"
+ },
+ {
+ "name":"b",
+ "type": {
+ "type":"array",
+ "items":{
+ "type":"record",
+ "name":"TestAllSupportedBaseTypes",
+ "fields":[
+ {
+ "name":"a",
+ "type": "boolean"
+ },
+ {
+ "name":"b",
+ "type":"int"
+ },
+ {
+ "name":"c",
+ "type":"int"
+ },
+ {
+ "name":"d",
+ "type":"int"
+ },
+ {
+ "name":"e",
+ "type":"int"
+ },
+ {
+ "name":"f",
+ "type":"int"
+ },
+ {
+ "name":"g",
+ "type":"long"
+ },
+ {
+ "name":"h",
+ "type":"float"
+ },
+ {
+ "name":"i",
+ "type":"double"
+ },
+ {
+ "name":"j",
+ "type":"string"
+ }
+ ]
+ }
+ }
+ },
+ {
+ "name":"c",
+ "type": {
+ "type":"map",
+ "values":"TestAllSupportedBaseTypes"
+ }
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(
+ schema,
+ TestGeneric::<TestAllSupportedBaseTypes>::get_schema()
+ );
+ let test_generic = TestGeneric::<TestAllSupportedBaseTypes> {
+ a: "testing".to_owned(),
+ b: vec![TestAllSupportedBaseTypes {
+ a: true,
+ b: 8_i8,
+ c: 16_i16,
+ d: 32_i32,
+ e: 8_u8,
+ f: 16_u16,
+ g: 64_i64,
+ h: 32.3333_f32,
+ i: 64.4444_f64,
+ j: "testing string".to_owned(),
+ }],
+ c: vec![(
+ "key".to_owned(),
+ TestAllSupportedBaseTypes {
+ a: true,
+ b: 8_i8,
+ c: 16_i16,
+ d: 32_i32,
+ e: 8_u8,
+ f: 16_u16,
+ g: 64_i64,
+ h: 32.3333_f32,
+ i: 64.4444_f64,
+ j: "testing string".to_owned(),
+ },
+ )]
+ .into_iter()
+ .collect(),
+ };
+ serde_assert(test_generic);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ enum TestAllowedEnum {
+ A,
+ B,
+ C,
+ D,
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestAllowedEnumNested {
+ a: TestAllowedEnum,
+ b: String,
+ }
+
+ #[test]
+ fn test_enum() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestAllowedEnumNested",
+ "fields":[
+ {
+ "name":"a",
+ "type": {
+ "type":"enum",
+ "name":"TestAllowedEnum",
+ "symbols":["A","B","C","D"]
+ }
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestAllowedEnumNested::get_schema());
+ let enum_included = TestAllowedEnumNested {
+ a: TestAllowedEnum::B,
+ b: "hey".to_owned(),
+ };
+ serde_assert(enum_included);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct ConsList {
+ value: i32,
+ next: Option<Box<ConsList>>,
+ }
+
+ #[test]
+ fn test_cons() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"ConsList",
+ "fields":[
+ {
+ "name":"value",
+ "type":"int"
+ },
+ {
+ "name":"next",
+ "type":["null","ConsList"]
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, ConsList::get_schema());
+ let list = ConsList {
+ value: 34,
+ next: Some(Box::new(ConsList {
+ value: 42,
+ next: None,
+ })),
+ };
+ serde_assert(list)
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct ConsListGeneric<T: AvroSchemaComponent> {
+ value: T,
+ next: Option<Box<ConsListGeneric<T>>>,
+ }
+
+ #[test]
+ fn test_cons_generic() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"ConsListGeneric",
+ "fields":[
+ {
+ "name":"value",
+ "type":{
+ "type":"record",
+ "name":"TestAllowedEnumNested",
+ "fields":[
+ {
+ "name":"a",
+ "type": {
+ "type":"enum",
+ "name":"TestAllowedEnum",
+ "symbols":["A","B","C","D"]
+ }
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ },
+ {
+ "name":"next",
+ "type":["null","ConsListGeneric"]
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(
+ schema,
+ ConsListGeneric::<TestAllowedEnumNested>::get_schema()
+ );
+ let list = ConsListGeneric::<TestAllowedEnumNested> {
+ value: TestAllowedEnumNested {
+ a: TestAllowedEnum::B,
+ b: "testing".into(),
+ },
+ next: Some(Box::new(ConsListGeneric::<TestAllowedEnumNested> {
+ value: TestAllowedEnumNested {
+ a: TestAllowedEnum::D,
+ b: "testing2".into(),
+ },
+ next: None,
+ })),
+ };
+ serde_assert(list)
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestSimpleArray {
+ a: [i32; 4],
+ }
+
+ #[test]
+ fn test_simple_array() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestSimpleArray",
+ "fields":[
+ {
+ "name":"a",
+ "type": {
+ "type":"array",
+ "items":"int"
+ }
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestSimpleArray::get_schema());
+ let test = TestSimpleArray { a: [2, 3, 4, 5] };
+ serde_assert(test)
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestComplexArray<T: AvroSchemaComponent> {
+ a: [T; 2],
+ }
+
+ #[test]
+ fn test_complex_array() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestComplexArray",
+ "fields":[
+ {
+ "name":"a",
+ "type": {
+ "type":"array",
+ "items":{
+ "type":"record",
+ "name":"TestBasic",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int"
+ },
+ {
+ "name":"b",
+ "type":"string"
+ }
+ ]
+ }
+ }
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestComplexArray::<TestBasic>::get_schema());
+ let test = TestComplexArray::<TestBasic> {
+ a: [
+ TestBasic {
+ a: 27,
+ b: "foo".to_owned(),
+ },
+ TestBasic {
+ a: 28,
+ b: "bar".to_owned(),
+ },
+ ],
+ };
+ serde_assert(test)
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct Testu8 {
+ a: Vec<u8>,
+ b: [u8; 2],
+ }
+ #[test]
+ fn test_bytes_handled() {
+ let test = Testu8 {
+ a: vec![1, 2],
+ b: [3, 4],
+ };
+ serde_assert(test)
+ // don't check for schema equality to allow for transitioning to bytes or fixed types in the future
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema)]
+ #[allow(unknown_lints)] // Rust 1.51.0 (MSRV) does not support #[allow(clippy::box_collection)]
+ #[allow(clippy::box_collection)]
+ struct TestSmartPointers<'a> {
+ a: Box<String>,
+ b: Mutex<Vec<i64>>,
+ c: Cow<'a, i32>,
+ }
+
+ #[test]
+ fn test_smart_pointers() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestSmartPointers",
+ "fields":[
+ {
+ "name":"a",
+ "type": "string"
+ },
+ {
+ "name":"b",
+ "type":{
+ "type":"array",
+ "items":"long"
+ }
+ },
+ {
+ "name":"c",
+ "type":"int"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestSmartPointers::get_schema());
+ let test = TestSmartPointers {
+ a: Box::new("hey".into()),
+ b: Mutex::new(vec![42]),
+ c: Cow::Owned(32),
+ };
+ // test serde with manual equality for mutex
+ let test = serde(test);
+ assert_eq!(Box::new("hey".into()), test.a);
+ assert_eq!(vec![42], *test.b.borrow().lock().unwrap());
+ assert_eq!(Cow::Owned::<i32>(32), test.c);
+ }
+
+ #[derive(Debug, Serialize, AvroSchema, Clone, PartialEq)]
+ struct TestReference<'a> {
+ a: &'a Vec<i32>,
+ b: &'static str,
+ c: &'a f64,
+ }
+
+ #[test]
+ fn test_reference_struct() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestReference",
+ "fields":[
+ {
+ "name":"a",
+ "type": {
+ "type":"array",
+ "items":"int"
+ }
+ },
+ {
+ "name":"b",
+ "type":"string"
+ },
+ {
+ "name":"c",
+ "type":"double"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ assert_eq!(schema, TestReference::get_schema());
+ let a = vec![34];
+ let c = 4.55555555_f64;
+ let test = TestReference {
+ a: &a,
+ b: "testing_static",
+ c: &c,
+ };
+ ser(test);
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ #[avro(namespace = "com.testing.namespace", doc = "A Documented Record")]
+ struct TestBasicWithAttributes {
+ #[avro(doc = "Milliseconds since Queen released Bohemian Rhapsody")]
+ a: i32,
+ #[avro(doc = "Full lyrics of Bohemian Rhapsody")]
+ b: String,
+ }
+
+ #[test]
+ fn test_basic_with_attributes() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"com.testing.namespace.TestBasicWithAttributes",
+ "doc":"A Documented Record",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int",
+ "doc":"Milliseconds since Queen released Bohemian Rhapsody"
+ },
+ {
+ "name":"b",
+ "type": "string",
+ "doc": "Full lyrics of Bohemian Rhapsody"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ if let Schema::Record { name, doc, .. } = TestBasicWithAttributes::get_schema() {
+ assert_eq!("com.testing.namespace".to_owned(), name.namespace.unwrap());
+ assert_eq!("A Documented Record", doc.unwrap())
+ } else {
+ panic!("TestBasicWithAttributes schema must be a record schema")
+ }
+ assert_eq!(schema, TestBasicWithAttributes::get_schema());
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ #[avro(namespace = "com.testing.namespace")]
+ /// A Documented Record
+ struct TestBasicWithOuterDocAttributes {
+ #[avro(doc = "Milliseconds since Queen released Bohemian Rhapsody")]
+ a: i32,
+ #[avro(doc = "Full lyrics of Bohemian Rhapsody")]
+ b: String,
+ }
+
+ #[test]
+ fn test_basic_with_out_doc_attributes() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"com.testing.namespace.TestBasicWithOuterDocAttributes",
+ "doc":"A Documented Record",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int",
+ "doc":"Milliseconds since Queen released Bohemian Rhapsody"
+ },
+ {
+ "name":"b",
+ "type": "string",
+ "doc": "Full lyrics of Bohemian Rhapsody"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ if let Schema::Record { name, doc, .. } = TestBasicWithOuterDocAttributes::get_schema() {
+ assert_eq!("com.testing.namespace".to_owned(), name.namespace.unwrap());
+ assert_eq!("A Documented Record", doc.unwrap())
+ } else {
+ panic!("TestBasicWithOuterDocAttributes schema must be a record schema")
+ }
+ assert_eq!(schema, TestBasicWithOuterDocAttributes::get_schema());
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ #[avro(namespace = "com.testing.namespace")]
+ /// A Documented Record
+ /// that spans
+ /// multiple lines
+ struct TestBasicWithLargeDoc {
+ #[avro(doc = "Milliseconds since Queen released Bohemian Rhapsody")]
+ a: i32,
+ #[avro(doc = "Full lyrics of Bohemian Rhapsody")]
+ b: String,
+ }
+
+ #[test]
+ fn test_basic_with_large_doc() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"com.testing.namespace.TestBasicWithLargeDoc",
+ "doc":"A Documented Record",
+ "fields":[
+ {
+ "name":"a",
+ "type":"int",
+ "doc":"Milliseconds since Queen released Bohemian Rhapsody"
+ },
+ {
+ "name":"b",
+ "type": "string",
+ "doc": "Full lyrics of Bohemian Rhapsody"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ if let Schema::Record { name, doc, .. } = TestBasicWithLargeDoc::get_schema() {
+ assert_eq!("com.testing.namespace".to_owned(), name.namespace.unwrap());
+ assert_eq!(
+ "A Documented Record\nthat spans\nmultiple lines",
+ doc.unwrap()
+ )
+ } else {
+ panic!("TestBasicWithLargeDoc schema must be a record schema")
+ }
+ assert_eq!(schema, TestBasicWithLargeDoc::get_schema());
+ }
+
+ #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
+ struct TestBasicWithU32 {
+ a: u32,
+ }
+
+ #[test]
+ fn test_basic_with_u32() {
+ let schema = r#"
+ {
+ "type":"record",
+ "name":"TestBasicWithU32",
+ "fields":[
+ {
+ "name":"a",
+ "type":"long"
+ }
+ ]
+ }
+ "#;
+ let schema = Schema::parse_str(schema).unwrap();
+ if let Schema::Record { name, .. } = TestBasicWithU32::get_schema() {
+ assert_eq!("TestBasicWithU32", name.fullname(None))
+ } else {
+ panic!("TestBasicWithU32 schema must be a record schema")
+ }
+ assert_eq!(schema, TestBasicWithU32::get_schema());
+
+ serde_assert(TestBasicWithU32 { a: u32::MAX });
+ serde_assert(TestBasicWithU32 { a: u32::MIN });
+ serde_assert(TestBasicWithU32 { a: 1_u32 });
+ }
+}