You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/23 05:22:43 UTC
[GitHub] [hudi] the-other-tim-brown opened a new pull request, #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
the-other-tim-brown opened a new pull request, #6761:
URL: https://github.com/apache/hudi/pull/6761
### Change Logs
If a user provides a recursive proto schema, it will fail when we write to parquet. We need to allow the user to specify how many levels of recursion they want before truncating the remaining data.
Main changes to existing code:
- ProtoClassBasedSchemaProvider tracks number of times a message descriptor is seen within a branch of the schema traversal
- once the number of times that descriptor is seen exceeds the user provided limit, set the field to preset record that will contain two fields: 1) the remaining data serialized as a proto byte array, 2) the descriptors full name for context about what is in that byte array
- Converting from a proto to an avro now accounts for this truncation of the input
### Impact
As part of this change, I needed to change how the namespace was set for the Records within the Avro schema. Since we cannot repeat the exact same namespace + name, I made the namespace the path within the schema being traversed so each instance of a recursive message class will have a unique full name.
Marking this as low risk since the protobuf support is scheduled for the 0.13.0 release
**Risk level: low **
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1257492160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
* 88992744447ba98b7e9355b12d41fe92d3252ff2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1258972880
CI is flaky due to unrelated issue. going ahead w/ merging. https://github.com/apache/hudi/pull/6801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1255815843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c1fa14fd6b422201093b552f4995f98eee84fb9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #6761:
URL: https://github.com/apache/hudi/pull/6761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1257495051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
* 88992744447ba98b7e9355b12d41fe92d3252ff2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r982708506
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +83,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
+ private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
+ private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Review Comment:
Yes, that's where the name of the type is coming through. Remember that the name of a record and the name of a field are different but both use "name" in the avro schema definition. Line 56 is the field name. The record name is more like a class name in java where it can be defined once and reused for different variables. You can see this down at [line 141](https://github.com/apache/hudi/pull/6761/files#diff-c4931766d3118fdebd1fb97fce4b5c36923f6bd66f6226adbfba1dd7ec2ddbd4R141) where the type is simply `"type" : [ "null", "org.apache.hudi.proto.recursion_overflow" ]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r980155086
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
Review Comment:
I did not mean to ask if we are testing recursion depth in this test. in general, do we have tests covering diff values or recursion depth elsewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1255847755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
}, {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c1fa14fd6b422201093b552f4995f98eee84fb9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610)
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1256859389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r979540417
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
Review Comment:
Not in this test class. This is just testing whether we can convert into a schema. The recursion depth only comes into play when creating the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1256890580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jinyius commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
jinyius commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r981404962
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +83,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
+ private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
+ private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Review Comment:
why rename the field?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -18,7 +18,10 @@
package org.apache.hudi.utilities.sources.helpers;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.utilities.test.proto.Child;
import org.apache.hudi.utilities.test.proto.Nested;
+import org.apache.hudi.utilities.test.proto.Parent;
Review Comment:
did you want to try using google's struct proto as a test case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1255844738
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
}, {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c1fa14fd6b422201093b552f4995f98eee84fb9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610)
* 510d525e15f1d334510109e70a416c42bf20a524 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jinyius commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
jinyius commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r982695954
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +83,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
+ private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
+ private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Review Comment:
L59? https://github.com/apache/hudi/pull/6761/files#diff-c4931766d3118fdebd1fb97fce4b5c36923f6bd66f6226adbfba1dd7ec2ddbd4R59
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1258318310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
}, {
"hash" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11744",
"triggerID" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88992744447ba98b7e9355b12d41fe92d3252ff2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727)
* a922a5beca9991c7b57e9640033075d5e289e5e2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r981791563
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +83,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
+ private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
+ private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Review Comment:
We're not renaming the field, it's just a name for the schema/type. You can see the output [here](https://github.com/apache/hudi/pull/6761/files#diff-c4931766d3118fdebd1fb97fce4b5c36923f6bd66f6226adbfba1dd7ec2ddbd4R56)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r979564092
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -141,24 +146,40 @@ private Schema getEnumSchema(Descriptors.EnumDescriptor enumDescriptor) {
return Schema.createEnum(enumDescriptor.getName(), null, getNamespace(enumDescriptor.getFullName()), symbols);
}
- private Schema getMessageSchema(Descriptors.Descriptor descriptor, Map<Descriptors.Descriptor, Schema> seen, boolean flattenWrappedPrimitives) {
- if (seen.containsKey(descriptor)) {
- return seen.get(descriptor);
+ /**
+ * Translates a Proto Message descriptor into an Avro Schema
+ * @param descriptor the descriptor for the proto message
+ * @param recursionDepths a map of the descriptor to the number of times it has been encountered in this depth first traversal of the schema.
+ * This is used to cap the number of times we recurse on a schema.
+ * @param flattenWrappedPrimitives if true, treat wrapped primitives as nullable primitives, if false, treat them as proto messages
+ * @param path a string prefixed with the namespace of the original message being translated to avro and containing the current dot separated path tracking progress through the schema.
+ * This value is used for a namespace when creating Avro records to avoid an error when reusing the same class name when unraveling a recursive schema.
+ * @param maxRecursionDepth the number of times to unravel a recursive proto schema before spilling the rest to bytes
+ * @return an avro schema
+ */
+ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
+ int maxRecursionDepth) {
+ // Parquet does not handle recursive schemas so we "unravel" the proto N levels
+ Integer currentRecursionCount = recursionDepths.getOrDefault(descriptor, 0);
+ if (currentRecursionCount >= maxRecursionDepth) {
+ return RECURSION_OVERFLOW_SCHEMA;
}
- Schema result = Schema.createRecord(descriptor.getName(), null,
- getNamespace(descriptor.getFullName()), false);
+ // The current path is used as a namespace to avoid record name collisions within recursive schemas
+ Schema result = Schema.createRecord(descriptor.getName(), null, path, false);
- seen.put(descriptor, result);
+ recursionDepths.put(descriptor, ++currentRecursionCount);
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, seen, flattenWrappedPrimitives), null, getDefault(f)));
+ // each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
Review Comment:
I think this will make the code harder to read. You would need to create copies of all entries related to your current path and add in the current field name to the path for your new keys and then re-lookup that path. I would prefer to leave this portion as is.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaWithOverflow(convertedSchema);
+ Parent input = inputAndOutput.getLeft();
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ // assert that overflow data can be read back into proto class
+ Child parsedSingleChildOverflow = Child.parseFrom(getOverflowBytesFromChildRecord((GenericRecord) actual.get("child")));
+ Assertions.assertEquals(input.getChild().getRecurseField().getRecurseField(), parsedSingleChildOverflow);
+ // Get children list
+ GenericData.Array<GenericRecord> array = (GenericData.Array<GenericRecord>) actual.get("children");
+ Child parsedChildren1Overflow = Child.parseFrom(getOverflowBytesFromChildRecord(array.get(0)));
+ Assertions.assertEquals(input.getChildren(0).getRecurseField().getRecurseField(), parsedChildren1Overflow);
+ Child parsedChildren2Overflow = Child.parseFrom(getOverflowBytesFromChildRecord(array.get(1)));
+ Assertions.assertEquals(input.getChildren(1).getRecurseField().getRecurseField(), parsedChildren2Overflow);
+ }
+
+ private Pair<Sample, GenericRecord> createInputOutputSampleWithWellKnownTypesNested(Schema schema) {
Review Comment:
Updating to make these values random
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1255813580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c1fa14fd6b422201093b552f4995f98eee84fb9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1257791778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88992744447ba98b7e9355b12d41fe92d3252ff2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1256092881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
}, {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c1fa14fd6b422201093b552f4995f98eee84fb9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610)
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1258636323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
}, {
"hash" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11744",
"triggerID" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a922a5beca9991c7b57e9640033075d5e289e5e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r980263321
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
Review Comment:
I'll add a test to the schema conversion that tests with the default depth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r980303555
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -141,24 +146,40 @@ private Schema getEnumSchema(Descriptors.EnumDescriptor enumDescriptor) {
return Schema.createEnum(enumDescriptor.getName(), null, getNamespace(enumDescriptor.getFullName()), symbols);
}
- private Schema getMessageSchema(Descriptors.Descriptor descriptor, Map<Descriptors.Descriptor, Schema> seen, boolean flattenWrappedPrimitives) {
- if (seen.containsKey(descriptor)) {
- return seen.get(descriptor);
+ /**
+ * Translates a Proto Message descriptor into an Avro Schema
+ * @param descriptor the descriptor for the proto message
+ * @param recursionDepths a map of the descriptor to the number of times it has been encountered in this depth first traversal of the schema.
+ * This is used to cap the number of times we recurse on a schema.
+ * @param flattenWrappedPrimitives if true, treat wrapped primitives as nullable primitives, if false, treat them as proto messages
+ * @param path a string prefixed with the namespace of the original message being translated to avro and containing the current dot separated path tracking progress through the schema.
+ * This value is used for a namespace when creating Avro records to avoid an error when reusing the same class name when unraveling a recursive schema.
+ * @param maxRecursionDepth the number of times to unravel a recursive proto schema before spilling the rest to bytes
+ * @return an avro schema
+ */
+ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
+ int maxRecursionDepth) {
+ // Parquet does not handle recursive schemas so we "unravel" the proto N levels
+ Integer currentRecursionCount = recursionDepths.getOrDefault(descriptor, 0);
+ if (currentRecursionCount >= maxRecursionDepth) {
+ return RECURSION_OVERFLOW_SCHEMA;
}
- Schema result = Schema.createRecord(descriptor.getName(), null,
- getNamespace(descriptor.getFullName()), false);
+ // The current path is used as a namespace to avoid record name collisions within recursive schemas
+ Schema result = Schema.createRecord(descriptor.getName(), null, path, false);
- seen.put(descriptor, result);
+ recursionDepths.put(descriptor, ++currentRecursionCount);
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, seen, flattenWrappedPrimitives), null, getDefault(f)));
+ // each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
Review Comment:
sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1256247487
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11610",
"triggerID" : "9c1fa14fd6b422201093b552f4995f98eee84fb9",
"triggerType" : "PUSH"
}, {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1256858000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 510d525e15f1d334510109e70a416c42bf20a524 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6761:
URL: https://github.com/apache/hudi/pull/6761#issuecomment-1258309575
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "510d525e15f1d334510109e70a416c42bf20a524",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11613",
"triggerID" : "510d525e15f1d334510109e70a416c42bf20a524",
"triggerType" : "PUSH"
}, {
"hash" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727",
"triggerID" : "88992744447ba98b7e9355b12d41fe92d3252ff2",
"triggerType" : "PUSH"
}, {
"hash" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a922a5beca9991c7b57e9640033075d5e289e5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88992744447ba98b7e9355b12d41fe92d3252ff2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11727)
* a922a5beca9991c7b57e9640033075d5e289e5e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r979482700
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -38,27 +39,43 @@ public class ProtoClassBasedSchemaProvider extends SchemaProvider {
* Configs supported.
*/
public static class Config {
- public static final String PROTO_SCHEMA_CLASS_NAME = "hoodie.deltastreamer.schemaprovider.proto.className";
- public static final String PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = "hoodie.deltastreamer.schemaprovider.proto.flattenWrappers";
+ private static final String PREFIX = "hoodie.deltastreamer.schemaprovider.proto";
+ public static final ConfigProperty<String> PROTO_SCHEMA_CLASS_NAME = ConfigProperty.key(PREFIX + ".className")
Review Comment:
usually we don't follow camel notations for configs.
PREFIX + ".class.name"
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -38,27 +39,43 @@ public class ProtoClassBasedSchemaProvider extends SchemaProvider {
* Configs supported.
*/
public static class Config {
- public static final String PROTO_SCHEMA_CLASS_NAME = "hoodie.deltastreamer.schemaprovider.proto.className";
- public static final String PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = "hoodie.deltastreamer.schemaprovider.proto.flattenWrappers";
+ private static final String PREFIX = "hoodie.deltastreamer.schemaprovider.proto";
+ public static final ConfigProperty<String> PROTO_SCHEMA_CLASS_NAME = ConfigProperty.key(PREFIX + ".className")
+ .noDefaultValue()
+ .sinceVersion("0.13.0")
+ .withDocumentation("The Protobuf Message class used as the source for the schema.");
+
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = ConfigProperty.key(PREFIX + ".flattenWrappers")
+ .defaultValue(false)
+ .sinceVersion("0.13.0")
+ .withDocumentation("When set to false wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
+ public static final ConfigProperty<Integer> PROTO_SCHEMA_MAX_RECURSION_DEPTH = ConfigProperty.key(PREFIX + ".maxRecursionDepth")
Review Comment:
minor. empty line between two configs.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaWithOverflow(convertedSchema);
+ Parent input = inputAndOutput.getLeft();
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ // assert that overflow data can be read back into proto class
+ Child parsedSingleChildOverflow = Child.parseFrom(getOverflowBytesFromChildRecord((GenericRecord) actual.get("child")));
+ Assertions.assertEquals(input.getChild().getRecurseField().getRecurseField(), parsedSingleChildOverflow);
+ // Get children list
+ GenericData.Array<GenericRecord> array = (GenericData.Array<GenericRecord>) actual.get("children");
+ Child parsedChildren1Overflow = Child.parseFrom(getOverflowBytesFromChildRecord(array.get(0)));
+ Assertions.assertEquals(input.getChildren(0).getRecurseField().getRecurseField(), parsedChildren1Overflow);
+ Child parsedChildren2Overflow = Child.parseFrom(getOverflowBytesFromChildRecord(array.get(1)));
+ Assertions.assertEquals(input.getChildren(1).getRecurseField().getRecurseField(), parsedChildren2Overflow);
+ }
+
+ private Pair<Sample, GenericRecord> createInputOutputSampleWithWellKnownTypesNested(Schema schema) {
Review Comment:
I see that this method populates only static values.
A: can we generate random dynamic values.
B. If its cumbersome, can we declare static variables and reuse rather than generating it again and again. I assume this is called from many places.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +82,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRINGS = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL = Schema.create(Schema.Type.NULL);
Review Comment:
NULL_SCHEMA
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesFlattened(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreFlattened() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_flattened.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesFlattened(convertedSchema), actual);
+ }
+
+ @Test
+ public void recursiveSchema_noOverflow() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/parent_schema_recursive.avsc"));
+ Pair<Parent, GenericRecord> inputAndOutput = createInputOutputForRecursiveSchemaNoOverflow(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void recursiveSchema_withOverflow() throws Exception {
Review Comment:
do we have tests for diff values of maxRecursionDepth. would be good to add one.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -38,27 +39,43 @@ public class ProtoClassBasedSchemaProvider extends SchemaProvider {
* Configs supported.
*/
public static class Config {
- public static final String PROTO_SCHEMA_CLASS_NAME = "hoodie.deltastreamer.schemaprovider.proto.className";
- public static final String PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = "hoodie.deltastreamer.schemaprovider.proto.flattenWrappers";
+ private static final String PREFIX = "hoodie.deltastreamer.schemaprovider.proto";
+ public static final ConfigProperty<String> PROTO_SCHEMA_CLASS_NAME = ConfigProperty.key(PREFIX + ".className")
+ .noDefaultValue()
+ .sinceVersion("0.13.0")
+ .withDocumentation("The Protobuf Message class used as the source for the schema.");
+
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = ConfigProperty.key(PREFIX + ".flattenWrappers")
+ .defaultValue(false)
+ .sinceVersion("0.13.0")
+ .withDocumentation("When set to false wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
+ public static final ConfigProperty<Integer> PROTO_SCHEMA_MAX_RECURSION_DEPTH = ConfigProperty.key(PREFIX + ".maxRecursionDepth")
Review Comment:
same here.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
Review Comment:
createDefaultOutputWithWellKnownTypesNested and createDefaultOutputWithWellKnownTypesNested has code reuse opportunity. can you add a private method and reuse in both.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +82,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRINGS = Schema.create(Schema.Type.STRING);
Review Comment:
minor. STRING_SCHEMA
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -38,27 +39,43 @@ public class ProtoClassBasedSchemaProvider extends SchemaProvider {
* Configs supported.
*/
public static class Config {
- public static final String PROTO_SCHEMA_CLASS_NAME = "hoodie.deltastreamer.schemaprovider.proto.className";
- public static final String PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = "hoodie.deltastreamer.schemaprovider.proto.flattenWrappers";
+ private static final String PREFIX = "hoodie.deltastreamer.schemaprovider.proto";
+ public static final ConfigProperty<String> PROTO_SCHEMA_CLASS_NAME = ConfigProperty.key(PREFIX + ".className")
+ .noDefaultValue()
+ .sinceVersion("0.13.0")
+ .withDocumentation("The Protobuf Message class used as the source for the schema.");
+
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = ConfigProperty.key(PREFIX + ".flattenWrappers")
Review Comment:
same comment as above.
`PREFIX + "flatten.wrappers"`
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -35,24 +38,99 @@
import com.google.protobuf.UInt64Value;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import com.google.protobuf.util.Timestamps;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Scanner;
import java.util.function.Function;
import java.util.stream.Collectors;
public class TestProtoConversionUtil {
@Test
- public void allFieldsSet_wellKnownTypesAreNested() {
+ public void allFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ Pair<Sample, GenericRecord> inputAndOutput = createInputOutputSampleWithWellKnownTypesNested(convertedSchema);
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, inputAndOutput.getLeft()), convertedSchema);
+ Assertions.assertEquals(inputAndOutput.getRight(), actual);
+ }
+
+ @Test
+ public void noFieldsSet_wellKnownTypesAreNested() throws IOException {
+ Sample sample = Sample.newBuilder().build();
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/sample_schema_nested.avsc"));
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, sample), convertedSchema);
+ Assertions.assertEquals(createDefaultOutputWithWellKnownTypesNested(convertedSchema), actual);
+ }
+
+ @Test
+ public void allFieldsSet_wellKnownTypesAreFlattened() throws IOException {
Review Comment:
same comment as above.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -141,24 +146,40 @@ private Schema getEnumSchema(Descriptors.EnumDescriptor enumDescriptor) {
return Schema.createEnum(enumDescriptor.getName(), null, getNamespace(enumDescriptor.getFullName()), symbols);
}
- private Schema getMessageSchema(Descriptors.Descriptor descriptor, Map<Descriptors.Descriptor, Schema> seen, boolean flattenWrappedPrimitives) {
- if (seen.containsKey(descriptor)) {
- return seen.get(descriptor);
+ /**
+ * Translates a Proto Message descriptor into an Avro Schema
+ * @param descriptor the descriptor for the proto message
+ * @param recursionDepths a map of the descriptor to the number of times it has been encountered in this depth first traversal of the schema.
+ * This is used to cap the number of times we recurse on a schema.
+ * @param flattenWrappedPrimitives if true, treat wrapped primitives as nullable primitives, if false, treat them as proto messages
+ * @param path a string prefixed with the namespace of the original message being translated to avro and containing the current dot separated path tracking progress through the schema.
+ * This value is used for a namespace when creating Avro records to avoid an error when reusing the same class name when unraveling a recursive schema.
+ * @param maxRecursionDepth the number of times to unravel a recursive proto schema before spilling the rest to bytes
+ * @return an avro schema
+ */
+ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
+ int maxRecursionDepth) {
+ // Parquet does not handle recursive schemas so we "unravel" the proto N levels
+ Integer currentRecursionCount = recursionDepths.getOrDefault(descriptor, 0);
+ if (currentRecursionCount >= maxRecursionDepth) {
+ return RECURSION_OVERFLOW_SCHEMA;
}
- Schema result = Schema.createRecord(descriptor.getName(), null,
- getNamespace(descriptor.getFullName()), false);
+ // The current path is used as a namespace to avoid record name collisions within recursive schemas
+ Schema result = Schema.createRecord(descriptor.getName(), null, path, false);
- seen.put(descriptor, result);
+ recursionDepths.put(descriptor, ++currentRecursionCount);
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, seen, flattenWrappedPrimitives), null, getDefault(f)));
+ // each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
Review Comment:
not too strong on the suggestion. will leave it to you to decide.
this may result in creating too many new hashmaps.
did you think of having just one hashmap, but modifying the key to prefix the path along w/ descriptor may be.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -38,27 +39,43 @@ public class ProtoClassBasedSchemaProvider extends SchemaProvider {
* Configs supported.
*/
public static class Config {
- public static final String PROTO_SCHEMA_CLASS_NAME = "hoodie.deltastreamer.schemaprovider.proto.className";
- public static final String PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = "hoodie.deltastreamer.schemaprovider.proto.flattenWrappers";
+ private static final String PREFIX = "hoodie.deltastreamer.schemaprovider.proto";
Review Comment:
minor: `PREFIX` -> `PROTO_SCHEMA_PROVIDER_PREFIX`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jinyius commented on a diff in pull request #6761: [HUDI-4904] Add support for unraveling proto schemas in ProtoClassBasedSchemaProvider
Posted by GitBox <gi...@apache.org>.
jinyius commented on code in PR #6761:
URL: https://github.com/apache/hudi/pull/6761#discussion_r982721631
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -80,17 +83,19 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
* 2. Convert directly from a protobuf {@link Message} to a {@link GenericRecord} while properly handling enums and wrapped primitives mentioned above.
*/
private static class AvroSupport {
+ private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
+ private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
+ private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
+ private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Review Comment:
ah, got it. thank you for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org