You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/27 03:31:43 UTC
[GitHub] [hudi] the-other-tim-brown opened a new pull request, #6806: [HUDI-4905] Improve type handling in proto schema conversion
the-other-tim-brown opened a new pull request, #6806:
URL: https://github.com/apache/hudi/pull/6806
### Change Logs
After discussing the current implementation with other Protobuf users, there were a few asks around handling some types better.
1. Handle unsigned longs as a decimal (backed by a fixed size byte array) in avro to allow users to query these fields within their lake without converting to an unsigned long after reading the data or when writing the query (currently unsigned long is just stored as a signed long)
2. Handle timestamps by converting them to a long with the timestamp-micros logical type in avro
3. Make fields within a `oneOf` nullable since only one of them will ever be set
4. Change the default settings for handling wrapped primitives to just treat them as nullable primitives
### Impact
Default settings are updated but this feature is not yet released (part of 0.13.0 release)
**Risk level: low **
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1271803780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
}, {
"hash" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12054",
"triggerID" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57f8b811946c7c013cbc31e9bd8db469f70cda2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036)
* 7d5b9dc0a9334b96adcdb8e17964f31944e94e91 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12054)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1268918966
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12006",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12005",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981805427
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -314,7 +351,16 @@ private Object convertObject(Schema schema, Object value) {
case ENUM:
return GenericData.get().createEnum(value.toString(), schema);
case FIXED:
- return GenericData.get().createFixed(null, ((GenericFixed) value).bytes(), schema);
+ if (value instanceof byte[]) {
+ return GenericData.get().createFixed(null, (byte[]) value, schema);
+ }
+ Object unsignedLongValue = value;
+ if (unsignedLongValue instanceof Message) {
+ // Unwrap UInt64Value
+ unsignedLongValue = getWrappedValue(unsignedLongValue);
+ }
+ // convert the long to its unsigned value
+ return DECIMAL_CONVERSION.toFixed(new BigDecimal(toUnsignedBigInteger((Long) unsignedLongValue)), schema, schema.getLogicalType());
Review Comment:
There's one that takes long
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -85,53 +121,58 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
private static class AvroSupport {
private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ // The max unsigned long value has 20 digits, so a decimal with precision 20 and scale 0 is required to represent all possible values.
+ // A byte array of length N can store at most floor(log_10(2^(8 × N - 1) - 1)) base 10 digits so we require N = 9.
+ private static final Schema UNSIGNED_LONG_SCHEMA = LogicalTypes.decimal(20).addToSchema(Schema.createFixed("unsigned_long", null, "org.apache.hudi.protos", 9));
+ private static final Conversions.DecimalConversion DECIMAL_CONVERSION = new Conversions.DecimalConversion();
private static final String OVERFLOW_DESCRIPTOR_FIELD_NAME = "descriptor_full_name";
private static final String OVERFLOW_BYTES_FIELD_NAME = "proto_bytes";
private static final Schema RECURSION_OVERFLOW_SCHEMA = Schema.createRecord("recursion_overflow", null, "org.apache.hudi.proto", false,
Arrays.asList(new Schema.Field(OVERFLOW_DESCRIPTOR_FIELD_NAME, STRING_SCHEMA, null, ""),
new Schema.Field(OVERFLOW_BYTES_FIELD_NAME, Schema.create(Schema.Type.BYTES), null, "".getBytes())));
- private static final AvroSupport INSTANCE = new AvroSupport();
// A cache of the proto class name paired with whether wrapped primitives should be flattened as the key and the generated avro schema as the value
private static final Map<SchemaCacheKey, Schema> SCHEMA_CACHE = new ConcurrentHashMap<>();
// A cache with a key as the pair target avro schema and the proto descriptor for the source and the value as an array of proto field descriptors where the order matches the avro ordering.
// When converting from proto to avro, we want to be able to iterate over the fields in the proto in the same order as they appear in the avro schema.
private static final Map<Pair<Schema, Descriptors.Descriptor>, Descriptors.FieldDescriptor[]> FIELD_CACHE = new ConcurrentHashMap<>();
- private static final Map<Descriptors.Descriptor, Schema.Type> WRAPPER_DESCRIPTORS_TO_TYPE = getWrapperDescriptorsToType();
-
- private static Map<Descriptors.Descriptor, Schema.Type> getWrapperDescriptorsToType() {
- Map<Descriptors.Descriptor, Schema.Type> wrapperDescriptorsToType = new HashMap<>();
- wrapperDescriptorsToType.put(StringValue.getDescriptor(), Schema.Type.STRING);
- wrapperDescriptorsToType.put(Int32Value.getDescriptor(), Schema.Type.INT);
- wrapperDescriptorsToType.put(UInt32Value.getDescriptor(), Schema.Type.INT);
- wrapperDescriptorsToType.put(Int64Value.getDescriptor(), Schema.Type.LONG);
- wrapperDescriptorsToType.put(UInt64Value.getDescriptor(), Schema.Type.LONG);
- wrapperDescriptorsToType.put(BoolValue.getDescriptor(), Schema.Type.BOOLEAN);
- wrapperDescriptorsToType.put(BytesValue.getDescriptor(), Schema.Type.BYTES);
- wrapperDescriptorsToType.put(DoubleValue.getDescriptor(), Schema.Type.DOUBLE);
- wrapperDescriptorsToType.put(FloatValue.getDescriptor(), Schema.Type.FLOAT);
- return wrapperDescriptorsToType;
- }
+ private static final Set<Descriptors.Descriptor> WRAPPER_DESCRIPTORS_TO_TYPE = getWrapperDescriptorsToType();
- private AvroSupport() {
+ private static Set<Descriptors.Descriptor> getWrapperDescriptorsToType() {
Review Comment:
We can inline this method like following:
```
val descriptors = new HashSet<>(Arrays.asList(...))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260460634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* fb2d9f00303ab0d955bb46f99bf1aefe02dc992f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1258928264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* ff5baa8706ecd246df835fd2c18f08057fa94dd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1258974989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* ff5baa8706ecd246df835fd2c18f08057fa94dd4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765)
* 0069da2d1acf2dbdd3e262e0347fe3224fb004e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r982614676
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
Review Comment:
All good, nvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
codope commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260424078
Canceling the CI run to prioritize release blockers. Apologies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260243069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* c5dff6337535dc26770b376383318065a8f4bd1d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776)
* eca2992d65733640a0999f29a5e35e6631bbc57a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803)
* fb2d9f00303ab0d955bb46f99bf1aefe02dc992f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r980743071
##########
hudi-utilities/src/test/resources/schema-provider/proto/sample_schema_defaults.avsc:
##########
@@ -0,0 +1,305 @@
+{
Review Comment:
Renamed from sample_schema_flattened.avsc
##########
hudi-utilities/src/test/resources/schema-provider/proto/sample_schema_wrapped_and_timestamp_as_record.avsc:
##########
@@ -0,0 +1,417 @@
+{
Review Comment:
Renamed from sample_schema_nested.avsc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1261173633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11848",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* fb2d9f00303ab0d955bb46f99bf1aefe02dc992f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260246561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* eca2992d65733640a0999f29a5e35e6631bbc57a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803)
* fb2d9f00303ab0d955bb46f99bf1aefe02dc992f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270621569
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r990227333
##########
hudi-utilities/pom.xml:
##########
@@ -85,7 +85,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
- <scope>test</scope>
Review Comment:
Yes, let's keep changes bounded -- shading is the right change but there's no reason to take it in this PR, since we're adding Proto deps to bundles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270775878
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r989232387
##########
hudi-utilities/pom.xml:
##########
@@ -85,7 +85,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
- <scope>test</scope>
Review Comment:
@alexeykudinkin and @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981730983
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -85,6 +93,8 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
private static class AvroSupport {
private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final Schema UNSIGNED_LONG_SCHEMA = LogicalTypes.decimal(21).addToSchema(Schema.createFixed("unsigned_long", null, "org.apache.hudi.protos", 9));
Review Comment:
Adding a description but this can be 20 since we're using scale 0 here. There are only 20 digits in the max unsigned long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r982618791
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -314,7 +351,16 @@ private Object convertObject(Schema schema, Object value) {
case ENUM:
return GenericData.get().createEnum(value.toString(), schema);
case FIXED:
- return GenericData.get().createFixed(null, ((GenericFixed) value).bytes(), schema);
+ if (value instanceof byte[]) {
+ return GenericData.get().createFixed(null, (byte[]) value, schema);
+ }
+ Object unsignedLongValue = value;
+ if (unsignedLongValue instanceof Message) {
+ // Unwrap UInt64Value
+ unsignedLongValue = getWrappedValue(unsignedLongValue);
+ }
+ // convert the long to its unsigned value
+ return DECIMAL_CONVERSION.toFixed(new BigDecimal(toUnsignedBigInteger((Long) unsignedLongValue)), schema, schema.getLogicalType());
Review Comment:
Sync'd up offline, `BigDecimal.valueOf` treats longs as signed and there's no way to treat as unsigned. Keeping as is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1261156714
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270782032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012)
* 57f8b811946c7c013cbc31e9bd8db469f70cda2a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270981845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57f8b811946c7c013cbc31e9bd8db469f70cda2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin merged pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin merged PR #6806:
URL: https://github.com/apache/hudi/pull/6806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1259017185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* 0069da2d1acf2dbdd3e262e0347fe3224fb004e1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769)
* c5dff6337535dc26770b376383318065a8f4bd1d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981654227
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -45,10 +45,15 @@ public static class Config {
.sinceVersion("0.13.0")
.withDocumentation("The Protobuf Message class used as the source for the schema.");
- public static final ConfigProperty<Boolean> PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = ConfigProperty.key(PROTO_SCHEMA_PROVIDER_PREFIX + ".flatten.wrappers")
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_WRAPPED_PRIMITIVES_AS_RECORDS = ConfigProperty.key(PROTO_SCHEMA_PROVIDER_PREFIX + ".flatten.wrappers")
.defaultValue(false)
.sinceVersion("0.13.0")
- .withDocumentation("When set to false wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
+ .withDocumentation("When set to true wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
Review Comment:
Let's call out what default behavior is (otherwise reader needs to do double negation to realize)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -45,10 +45,15 @@ public static class Config {
.sinceVersion("0.13.0")
.withDocumentation("The Protobuf Message class used as the source for the schema.");
- public static final ConfigProperty<Boolean> PROTO_SCHEMA_FLATTEN_WRAPPED_PRIMITIVES = ConfigProperty.key(PROTO_SCHEMA_PROVIDER_PREFIX + ".flatten.wrappers")
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_WRAPPED_PRIMITIVES_AS_RECORDS = ConfigProperty.key(PROTO_SCHEMA_PROVIDER_PREFIX + ".flatten.wrappers")
.defaultValue(false)
.sinceVersion("0.13.0")
- .withDocumentation("When set to false wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
+ .withDocumentation("When set to true wrapped primitives like Int64Value are translated to a record with a single 'value' field instead of simply a nullable value");
+
+ public static final ConfigProperty<Boolean> PROTO_SCHEMA_TIMESTAMPS_AS_RECORDS = ConfigProperty.key(PROTO_SCHEMA_PROVIDER_PREFIX + ".timestamps.as.records")
+ .defaultValue(false)
+ .sinceVersion("0.13.0")
+ .withDocumentation("When set to true Timestamp fields are translated to a record with a seconds and nanos field, instead of a long with the timestamp-micros logical type");
Review Comment:
Same here
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -58,12 +65,13 @@ public class ProtoConversionUtil {
/**
* Creates an Avro {@link Schema} for the provided class. Assumes that the class is a protobuf {@link Message}.
* @param clazz The protobuf class
- * @param flattenWrappedPrimitives set to true to treat wrapped primitives like nullable fields instead of nested messages.
+ * @param wrappedPrimitivesAsRecords set to true to treat wrapped primitives like record with a single "value" field instead of simply a nullable field
* @param maxRecursionDepth the number of times to unravel a recursive proto schema before spilling the rest to bytes
+ * @param timestampsAsRecords if true convert {@link Timestamp} to a Record with a seconds and nanos field, otherwise convert protobuf {@link Timestamp} to a long with the time-mircos logical type.
* @return An Avro schema
*/
- public static Schema getAvroSchemaForMessageClass(Class clazz, boolean flattenWrappedPrimitives, int maxRecursionDepth) {
- return AvroSupport.get().getSchema(clazz, flattenWrappedPrimitives, maxRecursionDepth);
+ public static Schema getAvroSchemaForMessageClass(Class clazz, boolean wrappedPrimitivesAsRecords, int maxRecursionDepth, boolean timestampsAsRecords) {
+ return AvroSupport.get().getSchema(clazz, wrappedPrimitivesAsRecords, maxRecursionDepth, timestampsAsRecords);
Review Comment:
Similar comment: instead of passing these as params:
- Make `AvroSupport` non-singleton
- Pass config into `AvroSupport` and init these as fields in there
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -204,34 +226,48 @@ private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Desc
return schemaFinalizer.apply(Schema.create(Schema.Type.INT));
case UINT32:
case INT64:
- case UINT64:
case SINT64:
case FIXED64:
case SFIXED64:
return schemaFinalizer.apply(Schema.create(Schema.Type.LONG));
+ case UINT64:
+ return schemaFinalizer.apply(UNSIGNED_LONG_SCHEMA);
case MESSAGE:
- String updatedPath = appendFieldNameToPath(path, f.getName());
- if (flattenWrappedPrimitives && WRAPPER_DESCRIPTORS_TO_TYPE.containsKey(f.getMessageType())) {
+ String updatedPath = appendFieldNameToPath(path, fieldDescriptor.getName());
+ if (!wrappedPrimitivesAsRecords && WRAPPER_DESCRIPTORS_TO_TYPE.contains(fieldDescriptor.getMessageType())) {
// all wrapper types have a single field, so we can get the first field in the message's schema
- return schemaFinalizer.apply(Schema.createUnion(Arrays.asList(NULL_SCHEMA, getFieldSchema(f.getMessageType().getFields().get(0), recursionDepths, flattenWrappedPrimitives, updatedPath,
- maxRecursionDepth))));
+ return schemaFinalizer.apply(makeSchemaNullable(getFieldSchema(fieldDescriptor.getMessageType().getFields().get(0), recursionDepths, wrappedPrimitivesAsRecords, updatedPath,
+ maxRecursionDepth, timestampsAsRecords)));
+ }
+ if (!timestampsAsRecords && Timestamp.getDescriptor().equals(fieldDescriptor.getMessageType())) {
+ // Handle timestamps as long with logical type
+ return schemaFinalizer.apply(makeSchemaNullable(LogicalTypes.timestampMicros().addToSchema(Schema.create(Schema.Type.LONG))));
Review Comment:
Same comment as above
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -314,7 +351,16 @@ private Object convertObject(Schema schema, Object value) {
case ENUM:
return GenericData.get().createEnum(value.toString(), schema);
case FIXED:
- return GenericData.get().createFixed(null, ((GenericFixed) value).bytes(), schema);
+ if (value instanceof byte[]) {
+ return GenericData.get().createFixed(null, (byte[]) value, schema);
+ }
+ Object unsignedLongValue = value;
+ if (unsignedLongValue instanceof Message) {
+ // Unwrap UInt64Value
+ unsignedLongValue = getWrappedValue(unsignedLongValue);
+ }
+ // convert the long to its unsigned value
+ return DECIMAL_CONVERSION.toFixed(new BigDecimal(toUnsignedBigInteger((Long) unsignedLongValue)), schema, schema.getLogicalType());
Review Comment:
There's `BigDecimal.valueOf` we can use
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestProtoConversionUtil.java:
##########
@@ -130,10 +141,43 @@ public void recursiveSchema_withOverflow() throws Exception {
Assertions.assertEquals(input.getChildren(1).getRecurseField().getRecurseField(), parsedChildren2Overflow);
}
+ @Test
+ public void oneOfSchema() throws IOException {
+ Schema.Parser parser = new Schema.Parser();
+ Schema convertedSchema = parser.parse(getClass().getClassLoader().getResourceAsStream("schema-provider/proto/oneof_schema.avsc"));
+ WithOneOf input = WithOneOf.newBuilder().setLong(32L).build();
+ GenericRecord actual = serializeAndDeserializeAvro(ProtoConversionUtil.convertToAvro(convertedSchema, input), convertedSchema);
+
+ GenericData.Record expectedRecord = new GenericData.Record(convertedSchema);
+ expectedRecord.put("int", null);
+ expectedRecord.put("long", 32L);
+ expectedRecord.put("message", null);
+ Assertions.assertEquals(expectedRecord, actual);
+ }
+
+ private void assertUnsignedLongCorrectness(Schema convertedSchema, Sample input, GenericRecord actual, boolean wellKnownTypesAsRecords) {
Review Comment:
Good indicator of whether method is coherent is how easy it to judge what it does by simply looking at its signature. For this one it's not that easy.
I'd suggest to split it actually in 2 and instead make them like following:
```
assert(Schema.Field fieldSchema, Long expectedValue, ... actual, ...)
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -85,6 +93,8 @@ public static GenericRecord convertToAvro(Schema schema, Message message) {
private static class AvroSupport {
private static final Schema STRING_SCHEMA = Schema.create(Schema.Type.STRING);
private static final Schema NULL_SCHEMA = Schema.create(Schema.Type.NULL);
+ private static final Schema UNSIGNED_LONG_SCHEMA = LogicalTypes.decimal(21).addToSchema(Schema.createFixed("unsigned_long", null, "org.apache.hudi.protos", 9));
Review Comment:
Please add a comment explaining where 21 is coming from
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
Review Comment:
Should we do an assertion instead making sure that passed in schema is nullable?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -58,12 +65,13 @@ public class ProtoConversionUtil {
/**
* Creates an Avro {@link Schema} for the provided class. Assumes that the class is a protobuf {@link Message}.
* @param clazz The protobuf class
- * @param flattenWrappedPrimitives set to true to treat wrapped primitives like nullable fields instead of nested messages.
+ * @param wrappedPrimitivesAsRecords set to true to treat wrapped primitives like record with a single "value" field instead of simply a nullable field
* @param maxRecursionDepth the number of times to unravel a recursive proto schema before spilling the rest to bytes
+ * @param timestampsAsRecords if true convert {@link Timestamp} to a Record with a seconds and nanos field, otherwise convert protobuf {@link Timestamp} to a long with the time-mircos logical type.
* @return An Avro schema
*/
- public static Schema getAvroSchemaForMessageClass(Class clazz, boolean flattenWrappedPrimitives, int maxRecursionDepth) {
- return AvroSupport.get().getSchema(clazz, flattenWrappedPrimitives, maxRecursionDepth);
+ public static Schema getAvroSchemaForMessageClass(Class clazz, boolean wrappedPrimitivesAsRecords, int maxRecursionDepth, boolean timestampsAsRecords) {
Review Comment:
Let's pass in config in here instead of individual values (it's not gonna scale well)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -204,34 +226,48 @@ private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Desc
return schemaFinalizer.apply(Schema.create(Schema.Type.INT));
case UINT32:
case INT64:
- case UINT64:
case SINT64:
case FIXED64:
case SFIXED64:
return schemaFinalizer.apply(Schema.create(Schema.Type.LONG));
+ case UINT64:
+ return schemaFinalizer.apply(UNSIGNED_LONG_SCHEMA);
case MESSAGE:
- String updatedPath = appendFieldNameToPath(path, f.getName());
- if (flattenWrappedPrimitives && WRAPPER_DESCRIPTORS_TO_TYPE.containsKey(f.getMessageType())) {
+ String updatedPath = appendFieldNameToPath(path, fieldDescriptor.getName());
+ if (!wrappedPrimitivesAsRecords && WRAPPER_DESCRIPTORS_TO_TYPE.contains(fieldDescriptor.getMessageType())) {
// all wrapper types have a single field, so we can get the first field in the message's schema
- return schemaFinalizer.apply(Schema.createUnion(Arrays.asList(NULL_SCHEMA, getFieldSchema(f.getMessageType().getFields().get(0), recursionDepths, flattenWrappedPrimitives, updatedPath,
- maxRecursionDepth))));
+ return schemaFinalizer.apply(makeSchemaNullable(getFieldSchema(fieldDescriptor.getMessageType().getFields().get(0), recursionDepths, wrappedPrimitivesAsRecords, updatedPath,
Review Comment:
Let's break this expression up (for readability, extracting val for schema)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
Review Comment:
Let's extract this finalizer as a static method to avoid gotchas w/ closures
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
+ updatedSchema = makeSchemaNullable(schema);
+ }
+ if (fieldDescriptor.isRepeated()) {
Review Comment:
Order of these conditionals should be reversed (repeated field w/in oneof). Let's add a test for that
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -314,7 +351,16 @@ private Object convertObject(Schema schema, Object value) {
case ENUM:
return GenericData.get().createEnum(value.toString(), schema);
case FIXED:
- return GenericData.get().createFixed(null, ((GenericFixed) value).bytes(), schema);
+ if (value instanceof byte[]) {
+ return GenericData.get().createFixed(null, (byte[]) value, schema);
+ }
+ Object unsignedLongValue = value;
+ if (unsignedLongValue instanceof Message) {
Review Comment:
We should make an assertion that the type is UInt64 indeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260109367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* c5dff6337535dc26770b376383318065a8f4bd1d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776)
* eca2992d65733640a0999f29a5e35e6631bbc57a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981744618
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -314,7 +351,16 @@ private Object convertObject(Schema schema, Object value) {
case ENUM:
return GenericData.get().createEnum(value.toString(), schema);
case FIXED:
- return GenericData.get().createFixed(null, ((GenericFixed) value).bytes(), schema);
+ if (value instanceof byte[]) {
+ return GenericData.get().createFixed(null, (byte[]) value, schema);
+ }
+ Object unsignedLongValue = value;
+ if (unsignedLongValue instanceof Message) {
+ // Unwrap UInt64Value
+ unsignedLongValue = getWrappedValue(unsignedLongValue);
+ }
+ // convert the long to its unsigned value
+ return DECIMAL_CONVERSION.toFixed(new BigDecimal(toUnsignedBigInteger((Long) unsignedLongValue)), schema, schema.getLogicalType());
Review Comment:
There is no public `valueOf` that takes in a BigInteger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270792039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012)
* 57f8b811946c7c013cbc31e9bd8db469f70cda2a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1259020601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* c5dff6337535dc26770b376383318065a8f4bd1d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1258925226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1258971981
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* ff5baa8706ecd246df835fd2c18f08057fa94dd4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765)
* 0069da2d1acf2dbdd3e262e0347fe3224fb004e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1258961960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* ff5baa8706ecd246df835fd2c18f08057fa94dd4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1261354748
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11803",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813",
"triggerID" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"triggerType" : "PUSH"
}, {
"hash" : "fb2d9f00303ab0d955bb46f99bf1aefe02dc992f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11848",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* fb2d9f00303ab0d955bb46f99bf1aefe02dc992f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11813) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981739787
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
+ updatedSchema = makeSchemaNullable(schema);
+ }
+ if (fieldDescriptor.isRepeated()) {
Review Comment:
You can't have a repeated field inside of a `oneOf` but I switched the order anyways
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r981750304
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -173,24 +184,35 @@ private Schema getMessageSchema(Descriptors.Descriptor descriptor, CopyOnWriteMa
List<Schema.Field> fields = new ArrayList<>(descriptor.getFields().size());
for (Descriptors.FieldDescriptor f : descriptor.getFields()) {
// each branch of the schema traversal requires its own recursion depth tracking so copy the recursionDepths map
- fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), flattenWrappedPrimitives, path, maxRecursionDepth), null, getDefault(f)));
+ fields.add(new Schema.Field(f.getName(), getFieldSchema(f, new CopyOnWriteMap<>(recursionDepths), wrappedPrimitivesAsRecords, path, maxRecursionDepth, timestampsAsRecords),
+ null, getDefault(f)));
}
result.setFields(fields);
return result;
}
- private Schema getFieldSchema(Descriptors.FieldDescriptor f, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean flattenWrappedPrimitives, String path,
- int maxRecursionDepth) {
- Function<Schema, Schema> schemaFinalizer = f.isRepeated() ? Schema::createArray : Function.identity();
- switch (f.getType()) {
+ private Schema getFieldSchema(Descriptors.FieldDescriptor fieldDescriptor, CopyOnWriteMap<Descriptors.Descriptor, Integer> recursionDepths, boolean wrappedPrimitivesAsRecords, String path,
+ int maxRecursionDepth, boolean timestampsAsRecords) {
+ Function<Schema, Schema> schemaFinalizer = schema -> {
+ Schema updatedSchema = schema;
+ // all fields in the oneof will be treated as nullable
+ if (fieldDescriptor.getContainingOneof() != null && !(schema.getType() == Schema.Type.UNION && schema.getTypes().get(0).getType() == Schema.Type.NULL)) {
Review Comment:
if you are referring to `schema.isNullable()`, that method does not exist in older avro versions that we support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1260105459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f18fff886e0509bed666404800694b8aff3bb98f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f18fff886e0509bed666404800694b8aff3bb98f",
"triggerType" : "PUSH"
}, {
"hash" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11765",
"triggerID" : "ff5baa8706ecd246df835fd2c18f08057fa94dd4",
"triggerType" : "PUSH"
}, {
"hash" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11769",
"triggerID" : "0069da2d1acf2dbdd3e262e0347fe3224fb004e1",
"triggerType" : "PUSH"
}, {
"hash" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776",
"triggerID" : "c5dff6337535dc26770b376383318065a8f4bd1d",
"triggerType" : "PUSH"
}, {
"hash" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eca2992d65733640a0999f29a5e35e6631bbc57a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f18fff886e0509bed666404800694b8aff3bb98f UNKNOWN
* c5dff6337535dc26770b376383318065a8f4bd1d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11776)
* eca2992d65733640a0999f29a5e35e6631bbc57a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #6806:
URL: https://github.com/apache/hudi/pull/6806#discussion_r988360500
##########
hudi-utilities/pom.xml:
##########
@@ -85,7 +85,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
- <scope>test</scope>
Review Comment:
Should I revert that last commit then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1271798724
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
}, {
"hash" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57f8b811946c7c013cbc31e9bd8db469f70cda2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036)
* 7d5b9dc0a9334b96adcdb8e17964f31944e94e91 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1271997791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12036",
"triggerID" : "57f8b811946c7c013cbc31e9bd8db469f70cda2a",
"triggerType" : "PUSH"
}, {
"hash" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12054",
"triggerID" : "7d5b9dc0a9334b96adcdb8e17964f31944e94e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d5b9dc0a9334b96adcdb8e17964f31944e94e91 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12054)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270632241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6806: [HUDI-4905] Improve type handling in proto schema conversion
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6806:
URL: https://github.com/apache/hudi/pull/6806#issuecomment-1270626983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012",
"triggerID" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"triggerType" : "PUSH"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1261156714",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1267716047",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1267716350",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1267756845",
"triggerType" : "MANUAL"
}, {
"hash" : "f03f9610cf4e2c490d33ca734ca9b3241b2be778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032",
"triggerID" : "1268384708",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f03f9610cf4e2c490d33ca734ca9b3241b2be778 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12012) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12032)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org