You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@avro.apache.org by "roofdiver (via GitHub)" <gi...@apache.org> on 2025/09/10 23:02:53 UTC
[I] Byte Array Deserialization [avro-rs]
roofdiver opened a new issue, #285:
URL: https://github.com/apache/avro-rs/issues/285
Hi y'all I'm using the de-serialization method where you build out structs for the fields you want to extract. This works great, but I am running into an issue when deserializing byte arrays in avro.
I have a struct like this
```
#[derive(Debug,Deserialize)]
struct Services{
port: i64,
transport: Option<String>,
service_name: Option<String>,
banner: Option<Vec<u8>>,
tls: TLS
}
```
and I'm de-serializing this way
`
...
let reader = Reader::new(file)?;
for value in reader {
let value = value?;
let record: Services = from_value::<Services>(&value)?;
println!("{:#?}",record)
}
`
I've confirmed my avro schema from the file as well
`
{
'default': None,
'name': 'banner',
'type': ['null', 'bytes']
}
`
No matter what I do I get this error.
`Error: Error { details: Failed to deserialize Avro value into value: Expected an Array or Union, but got: Bytes([0, 91, 0, 0, 0, 0, 0, 0]) }`
This leads me to believe that I'm supposed to use an avro built in type (Bytes?) to deserialize the byte array since it's not liking Vec<u8>. The issue there is that I'm seeing Bytes as an Enum of the apache_avro Value which doesn't implement Deserialize.
Would really appreciate anything pointing me in the right direction.
Thanks,
K
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3304565095
Ok so I went looking in the source code for a solution and I found that you can use
` #[serde(with = "apache_avro::serde_avro_bytes_opt")] ` in your struct and then de-serialization of byte arrays works.
Then you can do something like this
```
#[derive(Debug, Deserialize, Serialize, Encode, Decode, Clone)]
pub struct Example{
#[serde(with = "apache_avro::serde_avro_bytes_opt")]
pub data: Option<Vec<u8>>,
}
```
I would like to put it out that there that I searched pretty heavily and couldn't find this until I dug into the source code. I think it may be worth noting in the examples on the doc page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3304736939
Fair enough! What's the best preferred place to edit the documentation for this project? I can go ahead and add something in 😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3304769512
Ok! I updated my pull request to modify the readme file and put a section at the bottom for byte array deserialization. I also updated my unit tests with the solution so they don't fail and added some comments. Let me know if that works!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3282526162
I do see something similar you worked on here:
https://lists.apache.org/thread/547f66q0l42yw5r5rxtb84b65osqg2bw.
I think it's the same issue that was occuring with deserializing Decimal, except in this case it's Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3277436850
Could you please send a PR with a failing unit test?
It would help us to debug and fix it faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3304521178
Hi! @martin-g I've created this PR https://github.com/apache/avro-rs/pull/293
This is my first open source contribution (if this could even be considered that), so please let me know if I did anything incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "roofdiver (via GitHub)" <gi...@apache.org>.
roofdiver commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3282503411
Hi, this is for a super specific type of data, if I did a PR would I need to provide an example avro file that contains a byte array?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g closed issue #285: Byte Array Deserialization
URL: https://github.com/apache/avro-rs/issues/285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Byte Array Deserialization [avro-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on issue #285:
URL: https://github.com/apache/avro-rs/issues/285#issuecomment-3304611590
Great!
It's time for your second open source contribution! 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org