Re: [I] Enums don't serialize correctly [avro-rs]

["Kriskras99 (via GitHub)" <gi...@apache.org>]

Kriskras99 commented on issue #365:
URL: https://github.com/apache/avro-rs/issues/365#issuecomment-3730627801

   Over the past month I've done a lot of thinking about how to improve the enum support in Avro and I think I finally have something that can work. It's something that's workable in both the derive code and the general code, and remains compatible with current schemas.
   
   # Current state of Rust enum support in Avro
   
   ## Plain enums
   Plain enums are serialized as Avro enums (only enum support by `#[derive(AvroSchema)]`):
   ```rust
   pub enum Foo {
       A,
       B,
       C,
   }
   ```
   ```json
   {
     "name": "Foo",
     "type": "enum",
     "symbols": ["A", "B", "C"]
   }
   ```
   
   ## Data enums
   Data enums are serialized as a record with a discriminator field (Avro enum) and a value field (Avro union):
   ```rust
   pub struct Bar {
       integer: i32,
   }
   pub enum Foo {
       A {
           field: String,
       },
       B(Bar)
   }
   ```
   ```json
   {
     "name": "Foo",
     "type": "record",
     "fields": [
       {
         "name": "type",
         "type": {
           "type": "enum",
           "symbols": ["A", "B"]
         }
       },
       {
         "name": "value",
         "type": [
           {
             "type": "record",
             "fields": [
               {
                 "name": "field",
                 "type": "string"
               }
             ]
           },
           {
             "name": "Bar",
             "type": "record",
             "fields": [
               {
                 "name": "integer",
                 "type": "int"
               }
             ]
           }
         ]
       }
     ]
   }
   ```
   The advantage of this approach is that it works for enums where multiple variants of the same type. It does not currently
   work with mixed enums as a unit variant will always be encoded as an Avro enum.
   
   ## Options
   Options have special support in the encoding logic to always produce a bare union.
   ```rust
   type Foo = Option<String>;
   ```
   ```json
   [
     "null",
     "string"
   ]
   ```
   
   # Alternative representations
   These alternative representations support mixed enums.
   
   ## Bare union
   If all types are unique, then a regular union can be used:
   ```rust
   pub enum Foo {
       A {
           field: String,
       },
       B(Bar),
       C,
   }
   ```
   ```json
   [
     {
       "type": "record",
       "name": "A",
       "fields": [
         {
           "name": "field",
           "type": "string"
         }
       ]
     },
     {
       "name": "Bar",
       "type": "record",
       "fields": [
         {
           "name": "integer",
           "type": "int"
         }
       ]
     },
     "null"
   ]
   ```
   Using the variant name as the namespace can prevent collisions for named schema types, this however doesn't work for
   unnamed types.
   
   ## Union with a record for every variant
   ```rust
   pub enum Foo {
       A {
           field: String,
       },
       B(Bar),
       C,
   }
   ```
   ```json
   [
     {
       "type": "record",
       "name": "A",
       "fields": [
         {
           "name": "field",
           "type": "string"
         }
       ]
     },
     {
       "type": "record",
       "name": "B",
       "fields": [
         {
           "name": "inner",
           "type": {
             "name": "Bar",
             "type": "record",
             "fields": [
               {
                 "name": "integer",
                 "type": "int"
               }
             ]
           }
         }
       ]
     },
     {
       "type": "record",
       "name": "C",
       "fields": [
         {
           "name": "inner",
           "type": "null"
         }
       ]
     }
   ]
   ```
   This representation will always work and is always as efficient when binary encoded compared to the bare union. However, 
   the schema definition does become larger inflating the JSON but also the in-memory representation.
   
   # Proposal
   
   ## Deriving
   It would be good to broaden the support of enums we can (de)serialize, I suggest the following schema derive strategy:
   1. If it is a plain enum, emit an Avro enum
   2. If it is a mixed/data enum, try to emit an Avro union
   3. If that fails, emit an Avro record with a `type` and `value` field
   
   ## Encoding/decoding
   When encoding or decoding we would just look at the schema to see what is expected. This does mean changing the signature
   of `to_value(value: S) -> Result<Value, Error>` to `to_value(value: S, schema: &Schema) -> Result<Value, Error>`. We could
   also put a bound on `S` that it has to implement `AvroSchema`, but that's not good performance wise as it cannot be cached.
   
   P.S. While thinking about this problem, it occurred to me that it's possible that the `AvroSchema` derive can produce
   invalid schemas (fields with the same name because of a `rename`, `Option<T>` where `T`'s schema is `null`). Users who
   are creating a schema by hand (either completely or generating schemas in code) can of course also have this issue. I
   think it would be a good idea to add a `Schema::validate(&self) -> Result<(), Error>` function so we can validate the
   generated schema in the derive, and users can check their own schemas.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org