[GitHub] [beam] damondouglas commented on a diff in pull request #25927: FileReadSchemaTransform

["damondouglas (via GitHub)" <gi...@apache.org>]

damondouglas commented on code in PR #25927:
URL: https://github.com/apache/beam/pull/25927#discussion_r1146688175


##########
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeans.java:
##########
@@ -38,6 +39,28 @@ public class SchemaAwareJavaBeans {
 
   private static final DefaultSchemaProvider DEFAULT_SCHEMA_PROVIDER = new DefaultSchemaProvider();
 
+  /** Convenience method for {@link AllPrimitiveDataTypes} instantiation. */

Review Comment:
   ```
   /** Convenience method for {@link AvroPrimitiveDataTypes} instantiation. */
   ```



##########
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeans.java:
##########
@@ -270,6 +400,55 @@ public static SerializableFunction<Row, DoublyNestedDataTypes> doublyNestedDataT
     return DEFAULT_SCHEMA_PROVIDER.fromRowFunction(DOUBLY_NESTED_DATA_TYPES_TYPE_DESCRIPTOR);
   }
 
+  /**
+   * Contains all primitive Java types supported by Avro. The purpose of this class is to test
+   * schema-aware PTransforms with flat {@link Schema} {@link Row}s.
+   */
+  @DefaultSchema(AutoValueSchema.class)
+  @AutoValue
+  public abstract static class AvroPrimitiveDataTypes implements Serializable {

Review Comment:
   I'll let you decide based on my comment and I'll LGTM either way.
   
   What I like about AvroPrimitiveDataTypes is that it makes it easy to write tests against field types we know would be compatible with Avro encodings.  What I struggle with is:
   1. The name implies a coupling of data encoding though I get it
   2. Writing tests against AllPrimitiveDataTypes that fail is valuable information on how we expose FieldType
   3. Dealing with annoying FieldTypes that aren't compatible with a Schema related use case is valuable information to figure out how a real user of Beam would have to deal with this when they don't have the ability to change the original class but do have the ability to change the Schema using, for example, https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/schemas/transforms/DropFields.html.  I dealt with this problem when writing tests for CsvIO and it was valuable discovery.  Writing tests around this problem of incompatible fields provided for me a usability test not just does this code work test.
   
   Another alternative would be just to get rid of the Byte field in AllPrimitiveDataTypes but that would require you to refactor the tests that break as a result.
   
   This comment also applies to the other additions to SchemaAwareJavaBeans and I didn't comment on those individually as it would have been redundant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org