[GitHub] [pulsar] moonboots300 opened a new issue #4271: Pulsar Avro Schema Enum improvement request

[GitBox <gi...@apache.org>]

moonboots300 opened a new issue #4271: Pulsar Avro Schema Enum improvement request
URL: https://github.com/apache/pulsar/issues/4271
 
 
   **Is your feature request related to a problem? Please describe.**
   The pulsar API is not granular enough around Avro Schemas and Enums. There are currently 2 problems with the Pulsar Avro Schema resolution: 
   
   1.  Not able to set required=True for Enum types in python for an enum. 
   ```
   class Event(Enum):
       OPEN=0
       CLOSE=1
   
   class Test:
        event = Event #### NO way to pass in 'required=True' for enum.
   ```
   
   Desired Schema (with required enum):
   ```
   {
       "type": "record",
       "name": "Test",
       "namespace": "com.test",
       "fields": [ {
   	     "name": "event",
   		"type": {
   				"type": "enum",
   				"name": "Event",
   				"symbols": ["OPEN", "CLOSE"]
   			}
                }
       ]
   }
   ```
   Actual Schema - generated by AvroSchema(Test)
   ```
   {
       "type": "record",
       "name": "Test",
       "namespace": "com.test",
       "fields": [ {
               "name": "event",
               "type": **["null", {**
                       "type": "enum",
                       "name": "Event",
                       "symbols": ["OPEN", "CLOSE"]
                   }
               ]
           }
       ]
   }
   
   ```
   required=True, was implemented for all Python types except for Enum in https://github.com/apache/pulsar/pull/3526
   
   Was the functionality missed for Enums?
   
   2. In java, Building a Schema with Enums and withAlwaysAllowNull(false) is too restrictive. Suppose I have an AvroSchema that is defined by a java Class with two Enums, one that is required and one that is not required. 
   ```
   public class Payload{
        private Event event;
        private Direction direction;
   .....
        public enum Event { OPEN, CLOSE }
        public enum Direction { UP, DOWN }
   }
   ```
   
   Create an AvroSchema<Payload> - Neither of the following will give the desired Schema 
   ```
   AvroSchema<Payload> avroSchema = AvroSchema.of(SchemaDefinition.<Payload>builder().withPojo(Payload.class).**withAlwaysAllowNull(false)**.build());
   //OR 
   AvroSchema<Payload> avroSchema = AvroSchema.of(SchemaDefinition.<Payload>builder().withPojo(Payload.class).**withAlwaysAllowNull(true)**.build());
   //OR 
   AvroSchema<Payload> avroSchema = AvroSchema.of(SchemaDefinition.<Payload>builder().withPojo(Payload.class).build());
   ```
   **Desired Schema:** (One Nullable - event, and One Required - direction enum symbols)
   ```
   {
       "type": "record",
       "name": "Payload",
       "namespace": "com.test",
       "fields": [ {
   	     "name": "event",
   		"type": {
   				"type": "enum",
   				"name": "Event",
   				"symbols": ["OPEN", "CLOSE"]
   			}
                },
                {
   	     "name": "direction",
   		"type": ["null",{
   				"type": "enum",
   				"name": "Direction",
   				"symbols": ["UP", "DOWN"]
   			}]
                }
       ]
   }
   ```
   
   **Describe the solution you'd like**
   Allow the schema to be built from a String definition of the Schema in Java and Python. OR implement some method to identify each field as required or not. Python already has this, except it does **not** exist for enums. The Java implementation is too restrictive, requiring all of the Class fields to be non-null or allowing all fields to be null. There is no in-between (for there to be one Enum as Required and another as Nullable).
   
   **Describe alternatives you've considered**
   Using avro library instead of the wrapped pulsar functionality since it works for these cases.
   
   **Additional context**
   Add any other context or screenshots about the feature request here.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services