You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Rajiv M Ranganath <ra...@gmail.com> on 2022/08/29 07:04:32 UTC
Rust Serde Deserialize
Hi,
I am new to Avro. When I have an Arvo schema of the form,
```
record Abcd {
union { null, bytes } efgh = null;
}
```
What would be the corresponding Rust struct?
I tried
```
#[derive(Debug, Deserialize, PartialEq)]
struct Abcd {
efgh: Option<Vec<u8>>,
}
```
But for some reason, `apache_avro::from_value` is giving an
`Err(DeserializeValue("not an array"))` error.
Best,
Rajiv
Re: Rust Serde Deserialize
Posted by Martin Grigorov <mg...@apache.org>.
https://github.com/lerouxrgd/rsgen-avro/pull/36
On Tue, Aug 30, 2022 at 9:07 AM Martin Grigorov <mg...@apache.org>
wrote:
> Hi Rajiv,
>
> Now, when you said it I remembered about this prerequisite for serializing
> bytes.
> I will open a PR to rsgen-avro to add #[serde(with = "serde_bytes")] when
> "bytes" schema is used!
>
> Regards,
> Martin
>
> On Mon, Aug 29, 2022 at 5:10 PM Rajiv M Ranganath <
> rajiv.ranganath@gmail.com> wrote:
>
>> Hi Martin,
>>
>> It looks like there are no changes needed on the Avro side. I was not
>> using Serde correctly.
>>
>> On the Rust side, we need to use `serde_bytes` [1] crate, and define our
>> Rust struct as follows.
>>
>> ```
>> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
>> serde::Serialize)]
>> #[serde(default)]
>> pub struct Abcd {
>> #[serde(with = "serde_bytes")]
>> pub b: Option<Vec<u8>>,
>> }
>> ```
>>
>> That seems to make things work.
>>
>> Thanks again for the reply.
>>
>> Best,
>> Rajiv
>>
>> [1] https://docs.rs/serde_bytes/latest/serde_bytes/
>>
>> On Mon, Aug 29, 2022 at 5:18 PM Rajiv M Ranganath
>> <ra...@gmail.com> wrote:
>> >
>> > On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov <mg...@apache.org>
>> wrote:
>> >
>> > [...]
>> >
>> > > Do you want to contribute the code as a failing unit test in a Pull
>> > > Request ?
>> > > With a fix would be awesome!
>> >
>> > Okay. Please give me a couple of days. I'll investigate and open a PR.
>> >
>> > Best,
>> > Rajiv
>>
>
Re: Rust Serde Deserialize
Posted by Martin Grigorov <mg...@apache.org>.
Hi Rajiv,
Now, when you said it I remembered about this prerequisite for serializing
bytes.
I will open a PR to rsgen-avro to add #[serde(with = "serde_bytes")] when
"bytes" schema is used!
Regards,
Martin
On Mon, Aug 29, 2022 at 5:10 PM Rajiv M Ranganath <ra...@gmail.com>
wrote:
> Hi Martin,
>
> It looks like there are no changes needed on the Avro side. I was not
> using Serde correctly.
>
> On the Rust side, we need to use `serde_bytes` [1] crate, and define our
> Rust struct as follows.
>
> ```
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
> serde::Serialize)]
> #[serde(default)]
> pub struct Abcd {
> #[serde(with = "serde_bytes")]
> pub b: Option<Vec<u8>>,
> }
> ```
>
> That seems to make things work.
>
> Thanks again for the reply.
>
> Best,
> Rajiv
>
> [1] https://docs.rs/serde_bytes/latest/serde_bytes/
>
> On Mon, Aug 29, 2022 at 5:18 PM Rajiv M Ranganath
> <ra...@gmail.com> wrote:
> >
> > On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov <mg...@apache.org>
> wrote:
> >
> > [...]
> >
> > > Do you want to contribute the code as a failing unit test in a Pull
> > > Request ?
> > > With a fix would be awesome!
> >
> > Okay. Please give me a couple of days. I'll investigate and open a PR.
> >
> > Best,
> > Rajiv
>
Re: Rust Serde Deserialize
Posted by Rajiv M Ranganath <ra...@gmail.com>.
Hi Martin,
It looks like there are no changes needed on the Avro side. I was not
using Serde correctly.
On the Rust side, we need to use `serde_bytes` [1] crate, and define our
Rust struct as follows.
```
#[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)]
#[serde(default)]
pub struct Abcd {
#[serde(with = "serde_bytes")]
pub b: Option<Vec<u8>>,
}
```
That seems to make things work.
Thanks again for the reply.
Best,
Rajiv
[1] https://docs.rs/serde_bytes/latest/serde_bytes/
On Mon, Aug 29, 2022 at 5:18 PM Rajiv M Ranganath
<ra...@gmail.com> wrote:
>
> On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov <mg...@apache.org> wrote:
>
> [...]
>
> > Do you want to contribute the code as a failing unit test in a Pull
> > Request ?
> > With a fix would be awesome!
>
> Okay. Please give me a couple of days. I'll investigate and open a PR.
>
> Best,
> Rajiv
Re: Rust Serde Deserialize
Posted by Rajiv M Ranganath <ra...@gmail.com>.
On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov <mg...@apache.org> wrote:
[...]
> Do you want to contribute the code as a failing unit test in a Pull
> Request ?
> With a fix would be awesome!
Okay. Please give me a couple of days. I'll investigate and open a PR.
Best,
Rajiv
Re: Rust Serde Deserialize
Posted by Martin Grigorov <mg...@apache.org>.
Hi Rajiv,
There are 0 known issues for the Rust SDK! :-)
Do you want to contribute the code as a failing unit test in a Pull Request
?
With a fix would be awesome!
Thanks!
Martin
On Mon, Aug 29, 2022 at 12:11 PM Rajiv M Ranganath <
rajiv.ranganath@gmail.com> wrote:
> Hi Martin,
>
> On Mon, Aug 29, 2022 at 1:43 PM Martin Grigorov <mg...@apache.org>
> wrote:
>
> [...]
>
> > I'd recommend you this nice tool for generating Rust structs from Avro
> > schema: https://github.com/lerouxrgd/rsgen-avro
>
> Thanks for the reply and the pointer to `rsgen-avro`. :-)
>
> It looks like there seems to be an issue in the way Serde is serializing
> the schema.
>
> Here is the example:
>
> ```main.rs
> use apache_avro::types::{Record, Value};
> use apache_avro::Schema;
>
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
> serde::Serialize)]
> #[serde(default)]
> pub struct Abcd {
> pub b: Option<Vec<u8>>,
> }
>
> #[inline(always)]
> fn default_abcd_b() -> Option<Vec<u8>> {
> None
> }
>
> impl Default for Abcd {
> fn default() -> Abcd {
> Abcd {
> b: default_abcd_b(),
> }
> }
> }
>
> fn main() {
> let writers_schema = Schema::parse_str(
> r#"
> {
> "type": "record",
> "name": "Abcd",
> "fields": [
> {"name": "b", "type": ["null", "bytes"], "default": null}
> ]
> }
> "#,
> )
> .unwrap();
>
> let mut abcd_manual = Record::new(&writers_schema).unwrap();
> abcd_manual.put(
> "b",
> Value::Union(1,
> Box::new(Value::Bytes("hello_world".as_bytes().to_vec()))),
> );
>
> println!("{:?}", abcd_manual);
>
> let abcd_manual_bytes =
> apache_avro::to_avro_datum(&writers_schema, abcd_manual).unwrap();
> println!("{:?}", abcd_manual_bytes);
>
> let abcd_serde_value = apache_avro::to_value(Abcd {
> b: Some("hello_world".as_bytes().to_vec()),
> }).unwrap();
>
> println!("{:?}", abcd_serde_value);
>
> let abcd_serde_bytes = apache_avro::to_avro_datum(&writers_schema,
> abcd_serde_value);
>
> println!("{:?}", abcd_serde_bytes);
> }
> ```
>
> Rather than creating an Avro Value of the form,
>
> ```
> Record { fields: [("b", Union(1, Bytes([104, 101, 108, 108, 111, 95,
> 119, 111, 114, 108, 100])))], schema_lookup: {"b": 0} }
> ```
>
> Serde seems to be generating an Avro Value,
>
> ```
> Record([("b", Union(1, Array([Int(104), Int(101), Int(108), Int(108),
> Int(111), Int(95), Int(119), Int(111), Int(114), Int(108),
> Int(100)])))])
> ```
>
> which is causing the subsequent conversion to bytes to fail.
>
> I was wondering if this is a known issue?
>
> Best,
> Rajiv
>
Re: Rust Serde Deserialize
Posted by Rajiv M Ranganath <ra...@gmail.com>.
Hi Martin,
On Mon, Aug 29, 2022 at 1:43 PM Martin Grigorov <mg...@apache.org> wrote:
[...]
> I'd recommend you this nice tool for generating Rust structs from Avro
> schema: https://github.com/lerouxrgd/rsgen-avro
Thanks for the reply and the pointer to `rsgen-avro`. :-)
It looks like there seems to be an issue in the way Serde is serializing
the schema.
Here is the example:
```main.rs
use apache_avro::types::{Record, Value};
use apache_avro::Schema;
#[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)]
#[serde(default)]
pub struct Abcd {
pub b: Option<Vec<u8>>,
}
#[inline(always)]
fn default_abcd_b() -> Option<Vec<u8>> {
None
}
impl Default for Abcd {
fn default() -> Abcd {
Abcd {
b: default_abcd_b(),
}
}
}
fn main() {
let writers_schema = Schema::parse_str(
r#"
{
"type": "record",
"name": "Abcd",
"fields": [
{"name": "b", "type": ["null", "bytes"], "default": null}
]
}
"#,
)
.unwrap();
let mut abcd_manual = Record::new(&writers_schema).unwrap();
abcd_manual.put(
"b",
Value::Union(1,
Box::new(Value::Bytes("hello_world".as_bytes().to_vec()))),
);
println!("{:?}", abcd_manual);
let abcd_manual_bytes =
apache_avro::to_avro_datum(&writers_schema, abcd_manual).unwrap();
println!("{:?}", abcd_manual_bytes);
let abcd_serde_value = apache_avro::to_value(Abcd {
b: Some("hello_world".as_bytes().to_vec()),
}).unwrap();
println!("{:?}", abcd_serde_value);
let abcd_serde_bytes = apache_avro::to_avro_datum(&writers_schema,
abcd_serde_value);
println!("{:?}", abcd_serde_bytes);
}
```
Rather than creating an Avro Value of the form,
```
Record { fields: [("b", Union(1, Bytes([104, 101, 108, 108, 111, 95,
119, 111, 114, 108, 100])))], schema_lookup: {"b": 0} }
```
Serde seems to be generating an Avro Value,
```
Record([("b", Union(1, Array([Int(104), Int(101), Int(108), Int(108),
Int(111), Int(95), Int(119), Int(111), Int(114), Int(108),
Int(100)])))])
```
which is causing the subsequent conversion to bytes to fail.
I was wondering if this is a known issue?
Best,
Rajiv
Re: Rust Serde Deserialize
Posted by Martin Grigorov <mg...@apache.org>.
Hi Rajiv,
I'd recommend you this nice tool for generating Rust structs from Avro
schema: https://github.com/lerouxrgd/rsgen-avro
$ cat q.avsc
1 │ {
2 │ "type": "record",
3 │ "name": "Abcd",
4 │ "fields": [
5 │ {"name": "b", "type": ["null", "bytes"], "default": null}
6 │ ]
7 │ }
generates:
cat q.rs
1 │
2 │ #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
serde::Serialize)]
3 │ #[serde(default)]
4 │ pub struct Abcd {
5 │ pub b: Option<Vec<u8>>,
6 │ }
7 │
8 │ #[inline(always)]
9 │ fn default_abcd_b() -> Option<Vec<u8>> { None }
10 │
11 │ impl Default for Abcd {
12 │ fn default() -> Abcd {
13 │ Abcd {
14 │ b: default_abcd_b(),
15 │ }
16 │ }
17 │ }
On Mon, Aug 29, 2022 at 10:05 AM Rajiv M Ranganath <
rajiv.ranganath@gmail.com> wrote:
> Hi,
>
> I am new to Avro. When I have an Arvo schema of the form,
>
> ```
> record Abcd {
> union { null, bytes } efgh = null;
> }
> ```
>
> What would be the corresponding Rust struct?
>
> I tried
>
> ```
> #[derive(Debug, Deserialize, PartialEq)]
> struct Abcd {
> efgh: Option<Vec<u8>>,
> }
> ```
>
> But for some reason, `apache_avro::from_value` is giving an
> `Err(DeserializeValue("not an array"))` error.
>
> Best,
> Rajiv
>