You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/03 18:16:52 UTC
[GitHub] [beam] johnclara opened a new issue, #22574: [Bug]: Non-Option Avro Union with generated SpecificRecord
johnclara opened a new issue, #22574:
URL: https://github.com/apache/beam/issues/22574
### What happened?
Hello, I am new to beam so I am probably doing something wrong:
I am trying to create PCollections of a SpecificRecord which can convert between Rows and the java generated class.
For fields with unions that are non-optional, I am hitting exceptiosn:
Here is a test avro schema:
```
{
"name": "MultiUnionSpecificRecord",
"namespace": "specificrecord",
"type": "record",
"doc": "",
"fields": [
{
"name": "foo",
"type": [ "int", "string" ],
"doc": ""
}
]
}
```
Using this to generate the MultiUnionSpecificRecord class:
```
java -jar .avro-tools-1.9.0.jar compile -string -dateTimeLogicalTypeImpl joda schema \
src/test/resources/MultiUnionSpecificRecord.avsc \
src/test/java/specificrecord/
```
Running this test:
```
MultiUnionSpecificRecord record = MultiUnionSpecificRecord
.newBuilder()
.setFoo(0)
.build();
Pipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
PCollection<MultiUnionSpecificRecord> events = p.apply(
Create.<MultiUnionSpecificRecord>of(record)
.withSchema(
AvroUtils.toBeamSchema(specificrecord.MultiUnionSpecificRecord.getClassSchema()),
TypeDescriptor.of(specificrecord.MultiUnionSpecificRecord.class),
AvroUtils.getToRowFunction(specificrecord.MultiUnionSpecificRecord.class, specificrecord.MultiUnionSpecificRecord.getClassSchema()),
AvroUtils.getFromRowFunction(specificrecord.MultiUnionSpecificRecord.class)
)
);
System.out.println("Events Schema: " + events.getSchema());
p.run();
```
Hitting this exception:
```
class java.lang.String cannot be cast to class org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value (java.lang.String is in module java.base of loader 'bootstrap'; org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value is in unnamed module of loader 'app')
java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value (java.lang.String is in module java.base of loader 'bootstrap'; org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value is in unnamed module of loader 'app')
at org.apache.beam.sdk.schemas.GetterBasedSchemaProvider$RowValueGettersFactory$GetOneOf.convert(GetterBasedSchemaProvider.java:311)
at org.apache.beam.sdk.schemas.GetterBasedSchemaProvider$RowValueGettersFactory$Converter.get(GetterBasedSchemaProvider.java:346)
at org.apache.beam.sdk.values.RowWithGetters.getValue(RowWithGetters.java:71)
at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:325)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$ZBaqkdOF.encode(Unknown Source
```
Relevant versions:
Java 11
```
org.apache.beam:beam-sdks-java-core:2.40.0
...
org.apache.avro:avro:1.8.2 -> 1.9.2
```
### Issue Priority
Priority: 3
### Issue Component
Component: sdk-java-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnclara commented on issue #22574: [Bug]: Non-Option Avro Union with generated SpecificRecord
Posted by GitBox <gi...@apache.org>.
johnclara commented on issue #22574:
URL: https://github.com/apache/beam/issues/22574#issuecomment-1204514335
Following up on this, it looks like the AvroUtils.AvroSpecificRecordFieldValueTypeSupplier does not check the schema for a LogicalType with a OneOfType.IDENTIFIER in order to handle this case. The ProtoMessageSchema.ProtoClassFieldValueTypeSupplier does do something like this:
See AvroUtils: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L663
and ProtoClassFieldValueTypeSupplier: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnclara commented on issue #22574: [Bug]: Non-Option Avro Union with generated SpecificRecord
Posted by GitBox <gi...@apache.org>.
johnclara commented on issue #22574:
URL: https://github.com/apache/beam/issues/22574#issuecomment-1204532745
There seems to be a workaround by using the GenericRecord to and from row methods:
```
private static class MultiUnionSpecificRecordToRowFn implements SerializableFunction<MultiUnionSpecificRecord, Row> {
private final SerializableFunction<GenericRecord, Row> fn;
MultiUnionSpecificRecordToRowFn(SerializableFunction<GenericRecord, Row> fn) {
this.fn = fn;
}
@Override
public Row apply(MultiUnionSpecificRecord input) {
return fn.apply(input);
}
}
private static class RowToMultiUnionSpecificRecordFn implements SerializableFunction<Row, MultiUnionSpecificRecord> {
private final SerializableFunction<Row, GenericRecord> fn;
RowToMultiUnionSpecificRecordFn(SerializableFunction<Row, GenericRecord> fn) {
this.fn = fn;
}
@Override
public MultiUnionSpecificRecord apply(Row input) {
GenericRecord genericRecord = fn.apply(input);
return (MultiUnionSpecificRecord) SpecificData.get().deepCopy(MultiUnionSpecificRecord.getClassSchema(), genericRecord);
}
}
```
```
SerializableFunction<MultiUnionSpecificRecord, Row> toRowFunction = new MultiUnionSpecificRecordToRowFn(
AvroUtils.getGenericRecordToRowFunction(AvroUtils.toBeamSchema(MultiUnionSpecificRecord.getClassSchema()))
);
SerializableFunction<Row, MultiUnionSpecificRecord> fromRowFunction = new RowToMultiUnionSpecificRecordFn(
AvroUtils.getRowToGenericRecordFunction(MultiUnionSpecificRecord.getClassSchema())
);
PCollection<MultiUnionSpecificRecord> events = p.apply(
Create.<MultiUnionSpecificRecord>of(record)
.withSchema(
AvroUtils.toBeamSchema(specificrecord.MultiUnionSpecificRecord.getClassSchema()),
TypeDescriptor.of(specificrecord.MultiUnionSpecificRecord.class),
toRowFunction,
fromRowFunction
)
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org