You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "lerouxrgd (via GitHub)" <gi...@apache.org> on 2024/07/18 11:56:14 UTC
[PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
lerouxrgd opened a new pull request, #3027:
URL: https://github.com/apache/avro/pull/3027
<!--
*Thank you very much for contributing to Apache Avro - we are happy that you want to help us improve Avro. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Avro a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/AVRO/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "AVRO-XXXX: [component] Title of the pull request", where *AVRO-XXXX* should be replaced by the actual issue number.
The *component* is optional, but can help identify the correct reviewers faster: either the language ("java", "python") or subsystem such as "build" or "doc" are good candidates.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests. You can [build the entire project](https://github.com/apache/avro/blob/main/BUILD.md) or just the [language-specific SDK](https://avro.apache.org/project/how-to-contribute/#unit-tests).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Every commit message references Jira issues in their subject lines. In addition, commits follow the guidelines from [How to write a good git commit message](https://chris.beams.io/posts/git-commit/)
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
-->
## What is the purpose of the change
This is a follow up of GH-1900 which aims at improving bytes/fixed (de)serialization using [serde_bytes](https://docs.rs/serde_bytes/latest/serde_bytes/).
## Verifying this change
This change added tests and can be verified as follows:
- New unit tests have been added, they are based on the ones in
GH-1900
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? Rustdoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "lerouxrgd (via GitHub)" <gi...@apache.org>.
lerouxrgd commented on code in PR #3027:
URL: https://github.com/apache/avro/pull/3027#discussion_r1682802382
##########
lang/rust/Cargo.toml:
##########
@@ -34,7 +34,7 @@ authors = ["Apache Avro team <de...@avro.apache.org>"]
license = "Apache-2.0"
repository = "https://github.com/apache/avro"
edition = "2021"
-rust-version = "1.70.0"
+rust-version = "1.73.0"
Review Comment:
This is required by `std::thread_local!`'s `LocalKey` when wrapped in a `Cell` for having get/set methods available, as you can see [in the doc](https://doc.rust-lang.org/std/thread/struct.LocalKey.html#method.get). Bumping MSRV was actually advised by clippy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #3027:
URL: https://github.com/apache/avro/pull/3027#issuecomment-2236381110
Please run `cargo fmt --all` and push again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "lerouxrgd (via GitHub)" <gi...@apache.org>.
lerouxrgd commented on PR #3027:
URL: https://github.com/apache/avro/pull/3027#issuecomment-2237782945
@martin-g Actually I've realized that some extras are needed for handling types such as `Option<Vec<u8>>`, I will send a follow-up PR very soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #3027:
URL: https://github.com/apache/avro/pull/3027#issuecomment-2236547158
Thank you, @lerouxrgd !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g merged PR #3027:
URL: https://github.com/apache/avro/pull/3027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "lerouxrgd (via GitHub)" <gi...@apache.org>.
lerouxrgd commented on code in PR #3027:
URL: https://github.com/apache/avro/pull/3027#discussion_r1682811028
##########
lang/rust/avro/src/lib.rs:
##########
@@ -841,6 +841,7 @@
//!
mod bigdecimal;
+mod bytes;
Review Comment:
Indeed, that was my first guess. Then I realized that there is the ambiguous serialization to either `Value::Bytes` or `Value::Fixed` that couldn't be solved by just using `serde_bytes`. Therefore, I ended up using your approach of having a thread local variable to guide the (de)serlializer. With this final approach only `serde_avro_{bytes,fixed,slice}` need to be re-exported, as it is the case in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "lerouxrgd (via GitHub)" <gi...@apache.org>.
lerouxrgd commented on PR #3027:
URL: https://github.com/apache/avro/pull/3027#issuecomment-2236468908
> Please run cargo fmt --all and push again
Done, for some reason I had to use `cargo +nightly fmt --all` for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #3027:
URL: https://github.com/apache/avro/pull/3027#discussion_r1682746112
##########
lang/rust/Cargo.toml:
##########
@@ -34,7 +34,7 @@ authors = ["Apache Avro team <de...@avro.apache.org>"]
license = "Apache-2.0"
repository = "https://github.com/apache/avro"
edition = "2021"
-rust-version = "1.70.0"
+rust-version = "1.73.0"
Review Comment:
According to https://github.com/serde-rs/bytes/blob/0.11.15/Cargo.toml#L12 the MSRV for serde_bytes is 1.53.
Why do we need to bump to 1.73.0 here ?
##########
lang/rust/avro/src/bytes.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cell::Cell;
+
+thread_local! {
+ /// A thread local that is used to decide how to serialize Rust bytes into an Avro
+ /// `types::Value` of type bytes.
+ ///
+ /// Relies on the fact that serde's serialization process is single-threaded.
+ pub(crate) static SER_BYTES_TYPE: Cell<BytesType> = const { Cell::new(BytesType::Bytes) };
+
+ /// A thread local that is used to decide how to deserialize an Avro `types::Value`
+ /// of type bytes into Rust bytes.
+ ///
+ /// Relies on the fact that serde's deserialization process is single-threaded.
+ pub(crate) static DE_BYTES_BORROWED: Cell<bool> = const { Cell::new(false) };
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum BytesType {
+ Bytes,
+ Fixed,
+}
+
+/// Efficient (de)serialization of Avro bytes values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
+/// vec_field: Vec<u8>,
+///
+/// #[serde(with = "serde_avro_fixed")]
Review Comment:
Shall we leave only the field for `bytes` here ? Like the doc for `slice` below.
##########
lang/rust/avro/src/bytes.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cell::Cell;
+
+thread_local! {
+ /// A thread local that is used to decide how to serialize Rust bytes into an Avro
+ /// `types::Value` of type bytes.
+ ///
+ /// Relies on the fact that serde's serialization process is single-threaded.
+ pub(crate) static SER_BYTES_TYPE: Cell<BytesType> = const { Cell::new(BytesType::Bytes) };
+
+ /// A thread local that is used to decide how to deserialize an Avro `types::Value`
+ /// of type bytes into Rust bytes.
+ ///
+ /// Relies on the fact that serde's deserialization process is single-threaded.
+ pub(crate) static DE_BYTES_BORROWED: Cell<bool> = const { Cell::new(false) };
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum BytesType {
+ Bytes,
+ Fixed,
+}
+
+/// Efficient (de)serialization of Avro bytes values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
+/// vec_field: Vec<u8>,
+///
+/// #[serde(with = "serde_avro_fixed")]
+/// fixed_field: [u8; 6],
+/// }
+/// ```
+pub mod serde_avro_bytes {
+ use serde::{Deserializer, Serializer};
+
+ pub fn serialize<S: Serializer>(bytes: &[u8], serializer: S) -> Result<S::Ok, S::Error> {
+ serde_bytes::serialize(bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<Vec<u8>, D::Error> {
+ serde_bytes::deserialize(deserializer)
+ }
+}
+
+/// Efficient (de)serialization of Avro fixed values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
Review Comment:
Shall we leave only the field for `fixed` here ? Like the doc for `slice` below.
##########
lang/rust/avro/src/lib.rs:
##########
@@ -841,6 +841,7 @@
//!
mod bigdecimal;
+mod bytes;
Review Comment:
I recall you asked for some re-export related to bytes in rsgen-avro.
Do you still need it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "lerouxrgd (via GitHub)" <gi...@apache.org>.
lerouxrgd commented on code in PR #3027:
URL: https://github.com/apache/avro/pull/3027#discussion_r1682805030
##########
lang/rust/avro/src/bytes.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cell::Cell;
+
+thread_local! {
+ /// A thread local that is used to decide how to serialize Rust bytes into an Avro
+ /// `types::Value` of type bytes.
+ ///
+ /// Relies on the fact that serde's serialization process is single-threaded.
+ pub(crate) static SER_BYTES_TYPE: Cell<BytesType> = const { Cell::new(BytesType::Bytes) };
+
+ /// A thread local that is used to decide how to deserialize an Avro `types::Value`
+ /// of type bytes into Rust bytes.
+ ///
+ /// Relies on the fact that serde's deserialization process is single-threaded.
+ pub(crate) static DE_BYTES_BORROWED: Cell<bool> = const { Cell::new(false) };
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum BytesType {
+ Bytes,
+ Fixed,
+}
+
+/// Efficient (de)serialization of Avro bytes values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
+/// vec_field: Vec<u8>,
+///
+/// #[serde(with = "serde_avro_fixed")]
Review Comment:
I was hesitating, but I thought showcasing both in the same struct could help a user who would be discovering this feature.
##########
lang/rust/avro/src/bytes.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cell::Cell;
+
+thread_local! {
+ /// A thread local that is used to decide how to serialize Rust bytes into an Avro
+ /// `types::Value` of type bytes.
+ ///
+ /// Relies on the fact that serde's serialization process is single-threaded.
+ pub(crate) static SER_BYTES_TYPE: Cell<BytesType> = const { Cell::new(BytesType::Bytes) };
+
+ /// A thread local that is used to decide how to deserialize an Avro `types::Value`
+ /// of type bytes into Rust bytes.
+ ///
+ /// Relies on the fact that serde's deserialization process is single-threaded.
+ pub(crate) static DE_BYTES_BORROWED: Cell<bool> = const { Cell::new(false) };
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum BytesType {
+ Bytes,
+ Fixed,
+}
+
+/// Efficient (de)serialization of Avro bytes values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
+/// vec_field: Vec<u8>,
+///
+/// #[serde(with = "serde_avro_fixed")]
+/// fixed_field: [u8; 6],
+/// }
+/// ```
+pub mod serde_avro_bytes {
+ use serde::{Deserializer, Serializer};
+
+ pub fn serialize<S: Serializer>(bytes: &[u8], serializer: S) -> Result<S::Ok, S::Error> {
+ serde_bytes::serialize(bytes, serializer)
+ }
+
+ pub fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<Vec<u8>, D::Error> {
+ serde_bytes::deserialize(deserializer)
+ }
+}
+
+/// Efficient (de)serialization of Avro fixed values.
+///
+/// This module is intended to be used through the Serde `with` attribute. See below
+/// example:
+///
+/// ```rust
+/// use apache_avro::{serde_avro_bytes, serde_avro_fixed};
+/// use serde::{Deserialize, Serialize};
+///
+/// #[derive(Serialize, Deserialize)]
+/// struct StructWithBytes {
+/// #[serde(with = "serde_avro_bytes")]
Review Comment:
I was hesitating, but I thought showcasing both in the same struct could help a user who would be discovering this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-3631: [Rust] More efficient (de)serialization using serde_bytes [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #3027:
URL: https://github.com/apache/avro/pull/3027#discussion_r1682842169
##########
.github/workflows/test-lang-rust-ci-ARM.yml:
##########
@@ -31,7 +31,7 @@ permissions:
env:
RUSTFLAGS: -Dwarnings
- CARGO_REGISTRIES_CRATES_IO_PROTOCOL: 'git' # TODO: remove this env var once MSRV is 1.70.0+
+ CARGO_REGISTRIES_CRATES_IO_PROTOCOL: 'git' # TODO: remove this env var once MSRV is 1.73.0+
Review Comment:
```suggestion
```
##########
.github/workflows/test-lang-rust-ci.yml:
##########
@@ -31,7 +31,7 @@ permissions:
env:
RUSTFLAGS: -Dwarnings
- CARGO_REGISTRIES_CRATES_IO_PROTOCOL: 'git' # TODO: remove this env var once MSRV is 1.70.0+
+ CARGO_REGISTRIES_CRATES_IO_PROTOCOL: 'git' # TODO: remove this env var once MSRV is 1.73.0+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org