[GitHub] [beam] ahmedabu98 commented on a diff in pull request #25927: FileReadSchemaTransform

["ahmedabu98 (via GitHub)" <gi...@apache.org>]

ahmedabu98 commented on code in PR #25927:
URL: https://github.com/apache/beam/pull/25927#discussion_r1147523858


##########
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeans.java:
##########
@@ -270,6 +400,55 @@ public static SerializableFunction<Row, DoublyNestedDataTypes> doublyNestedDataT
     return DEFAULT_SCHEMA_PROVIDER.fromRowFunction(DOUBLY_NESTED_DATA_TYPES_TYPE_DESCRIPTOR);
   }
 
+  /**
+   * Contains all primitive Java types supported by Avro. The purpose of this class is to test
+   * schema-aware PTransforms with flat {@link Schema} {@link Row}s.
+   */
+  @DefaultSchema(AutoValueSchema.class)
+  @AutoValue
+  public abstract static class AvroPrimitiveDataTypes implements Serializable {

Review Comment:
   Yeah the problem is Short (int16)  and Byte (a single Byte) are not types that are supported by Avro (see [Avro primitive types](https://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/avroschemas.html#:~:text=in%20this%20chapter.-,Primitive%20Data%20Types,-In%20the%20previous)) or Parquet (see [primitive types](https://parquet.apache.org/docs/file-format/types/)). So actually in AvroUtils we turn Short and Byte types into Beam Schema types to an Integer (int32).
   
   Avro and Parquet do support a sequence of bytes though, so I've replaced Byte with ByteBuffer in `AvroPrimitiveDataTypes`.
   
   >Another alternative would be just to get rid of the Byte field in AllPrimitiveDataTypes
   
   If you agree, I'd be happy to remove Short and replace Byte with ByteBuffer in `AllPrimitiveDataTypes` and use that for everything instead (for both read and write). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org