You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by GitBox <gi...@apache.org> on 2022/01/13 13:57:16 UTC
[GitHub] [avro] martin-g opened a new pull request #1456: AVRO-3302: Add interop tests for rust
martin-g opened a new pull request #1456:
URL: https://github.com/apache/avro/pull/1456
### Jira
- [X] My PR addresses the following [AVRO-3302](https://issues.apache.org/jira/browse/AVRO-3302) issue.
### Tests
- [X] My PR updates some unit tests
- [ ] Add a new unit test for the recursive schema with back-tracing
### Commits
- [X] My commits all reference Jira issues in their subject lines.
### Documentation
- [ ] Need to add inline comments for the logic with the RecordField#schema replacement for the recursive schema with backtracking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1016141787
Tested https://github.com/jorgecarleitao/arrow2.git:
```diff
diff --git Cargo.toml Cargo.toml
index 1803bea47..a664f76a3 100644
--- Cargo.toml
+++ Cargo.toml
@@ -95,7 +95,8 @@ tokio = { version = "1", features = ["macros", "rt", "fs"] }
tokio-util = { version = "0.6", features = ["compat"] }
# used to run formal property testing
proptest = { version = "1", default_features = false, features = ["std"] }
-avro-rs = { version = "0.13", features = ["snappy"] }
+# avro-rs = { version = "0.13", features = ["snappy"] }
+avro-rs = { git = "https://github.com/martin-g/avro", branch="avro-3302-add-interop-tests-for-rust", features = ["snappy"] }
```
```
$ ARROW2_IGNORE_PARQUET= cargo test --features full
```
No failures!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g edited a comment on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g edited a comment on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1015191375
> Thank you, @gklijs !
> I am going to use your library to test for regressions!
I've just tested (`cargo test --features avro,easy`) `schema_registry_converter` with the following patch and all tests still pass:
```diff
diff --git Cargo.toml Cargo.toml
index dc22fcc..953588e 100644
--- Cargo.toml
+++ Cargo.toml
@@ -44,7 +44,9 @@ features = ["derive"]
version = "^1.0"
[dependencies.avro-rs]
-version = "^0.13"
+# version = "^0.13"
+git = "https://github.com/martin-g/avro"
+branch="avro-3302-add-interop-tests-for-rust"
optional = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] gklijs commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
gklijs commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1014334255
Thanks a lot, nice to see the Rust side maturing, especially as the maintainer of a library that has it as a dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1016149568
Tested https://github.com/lerouxrgd/rsgen-avro.git:
```diff
diff --git Cargo.toml Cargo.toml
index 7812704..64fd6cd 100644
--- Cargo.toml
+++ Cargo.toml
@@ -10,7 +10,7 @@ license = "MIT"
readme = "README.md"
[dependencies]
-avro-rs = "0.13"
+avro-rs = { git = "https://github.com/martin-g/avro", branch="avro-3302-add-interop-tests-for-rust" }
docopt = "1"
glob = "0.3"
heck = "0.4"
diff --git src/templates.rs src/templates.rs
index 1d7f80b..17fa496 100644
--- src/templates.rs
+++ src/templates.rs
@@ -238,6 +238,7 @@ impl Templater {
if let Schema::Fixed {
name: Name { name, .. },
size,
+ ..
} = schema
{
let mut ctx = Context::new();
@@ -316,6 +317,7 @@ impl Templater {
o.insert(name_std.clone(), name);
match schema {
+ Schema::Ref { name: _ } => { todo!() }
Schema::Boolean => {
let default = self.parse_default(schema, gen_state, default.as_ref())?;
f.push(name_std.clone());
@@ -496,6 +498,7 @@ impl Templater {
let mut visitors = vec![];
for sc in schemas {
let symbol_str = match sc {
+ Schema::Ref { name: _ } => todo!(),
Schema::Boolean => "Boolean(bool)".into(),
Schema::Int => "Int(i32)".into(),
Schema::Long => "Long(i64)".into(),
@@ -595,6 +598,7 @@ impl Templater {
default: Option<&serde_json::Value>,
) -> Result<String> {
let default_str = match schema {
+ Schema::Ref { name: _ } => todo!(),
Schema::Boolean => match default {
Some(Value::Bool(b)) => b.to_string(),
None => bool::default().to_string(),
@@ -692,6 +696,7 @@ impl Templater {
Schema::Fixed {
name: Name { name: f_name, .. },
size,
+ ..
} => match default {
Some(Value::String(s)) => {
let bytes = s.clone().into_bytes();
@@ -709,6 +714,7 @@ impl Templater {
Schema::Fixed {
name: Name { name: f_name, .. },
size,
+ ..
} => match default {
Some(Value::String(s)) => {
let bytes = s.clone().into_bytes();
@@ -937,6 +943,10 @@ pub(crate) fn array_type(inner: &Schema, gen_state: &GenState) -> Result<String>
| Schema::Enum {
name: Name { name, .. },
..
+ }
+ | Schema::Ref {
+ name: Name { name, .. },
+ ..
} => format!("Vec<{}>", &sanitize(name.to_upper_camel_case())),
Schema::Null => err!("Invalid use of Schema::Null")?,
@@ -951,6 +961,7 @@ pub(crate) fn map_type(inner: &Schema, gen_state: &GenState) -> Result<String> {
}
let type_str = match inner {
+ Schema::Ref { name: _ } => map_of("String"),
Schema::Boolean => map_of("bool"),
Schema::Int => map_of("i32"),
Schema::Long => map_of("i64"),
@@ -1003,6 +1014,7 @@ pub(crate) fn map_type(inner: &Schema, gen_state: &GenState) -> Result<String> {
fn union_enum_variant(schema: &Schema, gen_state: &GenState) -> Result<String> {
let variant_str = match schema {
+ Schema::Ref { name: _ } => "String".into(),
Schema::Boolean => "Boolean".into(),
Schema::Int => "Int".into(),
Schema::Long => "Long".into(),
@@ -1080,6 +1092,7 @@ pub(crate) fn union_type(
/// Generates the Rust type of the inner schema of an Avro optional union.
pub(crate) fn option_type(inner: &Schema, gen_state: &GenState) -> Result<String> {
let type_str = match inner {
+ Schema::Ref { name: _ } => "Option<String>".into(),
Schema::Boolean => "Option<bool>".into(),
Schema::Int => "Option<i32>".into(),
```
```
$ cargo test
```
No failures!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1016134355
Tested arrow-datafusion:
```diff
diff --git datafusion/Cargo.toml datafusion/Cargo.toml
index d5334130..bf1634d4 100644
--- datafusion/Cargo.toml
+++ datafusion/Cargo.toml
@@ -74,7 +74,8 @@ regex = { version = "^1.4.3", optional = true }
lazy_static = { version = "^1.4.0" }
smallvec = { version = "1.6", features = ["union"] }
rand = "0.8"
-avro-rs = { version = "0.13", features = ["snappy"], optional = true }
+#avro-rs = { version = "0.13", features = ["snappy"], optional = true }
+avro-rs = { git = "https://github.com/martin-g/avro", branch="avro-3302-add-interop-tests-for-rust", features = ["snappy"], optional = true }
num-traits = { version = "0.2", optional = true }
pyo3 = { version = "0.14", optional = true }
tempfile = "3"
diff --git datafusion/src/avro_to_arrow/schema.rs datafusion/src/avro_to_arrow/schema.rs
index 2e9a17de..fd0ffc31 100644
--- datafusion/src/avro_to_arrow/schema.rs
+++ datafusion/src/avro_to_arrow/schema.rs
@@ -61,6 +61,7 @@ fn schema_to_field_with_props(
) -> Result<Field> {
let mut nullable = nullable;
let field_type: DataType = match schema {
+ AvroSchema::Ref { name: _ }=> todo!(),
AvroSchema::Null => DataType::Null,
AvroSchema::Boolean => DataType::Boolean,
AvroSchema::Int => DataType::Int32,
```
```
arrow-datafusion on master [!] via 🦀 v1.58.0 on ☁️ eu-central-1 took 10s
❯ cargo run --example avro_sql --features=datafusion/avro
Compiling rustversion v1.0.6
Compiling strum_macros v0.23.1
Compiling avro-rs v0.14.0 (https://github.com/martin-g/avro?branch=avro-3302-add-interop-tests-for-rust#6ee4d8ee)
Compiling datafusion v6.0.0 (/home/martin/git/rust/avro_regression_tests/arrow-datafusion/datafusion)
Compiling datafusion-examples v5.0.0 (/home/martin/git/rust/avro_regression_tests/arrow-datafusion/datafusion-examples)
Finished dev [unoptimized + debuginfo] target(s) in 45.93s
Running `target/debug/examples/avro_sql`
+---------+------------+----------------------------------------------+
| int_col | double_col | CAST(alltypes_plain.date_string_col AS Utf8) |
+---------+------------+----------------------------------------------+
| 1 | 10.1 | 03/01/09 |
| 1 | 10.1 | 04/01/09 |
| 1 | 10.1 | 02/01/09 |
+---------+------------+----------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g edited a comment on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g edited a comment on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1015191375
> Thank you, @gklijs !
> I am going to use your library to test for regressions!
I've just tested `schema_registry_converter` with the following patch and all tests still pass:
```diff
diff --git Cargo.toml Cargo.toml
index dc22fcc..953588e 100644
--- Cargo.toml
+++ Cargo.toml
@@ -44,7 +44,9 @@ features = ["derive"]
version = "^1.0"
[dependencies.avro-rs]
-version = "^0.13"
+# version = "^0.13"
+git = "https://github.com/martin-g/avro"
+branch="avro-3302-add-interop-tests-for-rust"
optional = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g merged pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g merged pull request #1456:
URL: https://github.com/apache/avro/pull/1456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1014356663
Thank you, @gklijs !
I am going to use your library to test for regressions!
I am working on a rather big refactor of this PR to add support for cyclic schema references. I hope my current approach will work because I have no more ideas how to do it in Rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] martin-g commented on pull request #1456: AVRO-3302: Add interop tests for rust
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #1456:
URL: https://github.com/apache/avro/pull/1456#issuecomment-1015191375
> Thank you, @gklijs !
> I am going to use your library to test for regressions!
I've just tested `schema_registry_converter` with the following patch and all tests still pass:
```
diff --git Cargo.toml Cargo.toml
index dc22fcc..953588e 100644
--- Cargo.toml
+++ Cargo.toml
@@ -44,7 +44,9 @@ features = ["derive"]
version = "^1.0"
[dependencies.avro-rs]
-version = "^0.13"
+# version = "^0.13"
+git = "https://github.com/martin-g/avro"
+branch="avro-3302-add-interop-tests-for-rust"
optional = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org