You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2024/04/22 20:04:34 UTC
[PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
the-other-tim-brown opened a new pull request, #11070:
URL: https://github.com/apache/hudi/pull/11070
### Change Logs
- Adds support for deserializing with the Confluent proto deserializer
- Adds convert to convert response from SchemaRegistryProvider to Avro for Proto topics
- Updates testing for these cases
### Impact
Allows users currently using Confluent + Proto to integrate with the Hudi Streamer more easily
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2102916239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d916b8da0cc796746409ce0512f9b80bc4d2205 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788)
* 4e15df959494651d74e92f6a998d3310bfc91247 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105735707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
}, {
"hash" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23854",
"triggerID" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851)
* 36cecc155c8cf9fbec29f895eea6111150eacf0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23854)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2073802830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c250cc04340a016a04878a7647d4b27a608e7374 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403)
* 1ce1316840852fa8e21363100f6ce695a5ecf0a7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2102800156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7ef574e9785174f786bb53c3c8c84cf6514da5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776)
* 2d916b8da0cc796746409ce0512f9b80bc4d2205 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2075504737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ce1316840852fa8e21363100f6ce695a5ecf0a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435)
* 04b49276121b062b285e1004bc72d779a54540d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1595524668
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -181,7 +218,7 @@ private static List<Sample> createSampleMessages(int count) {
.setWrappedDouble(DoubleValue.of(RANDOM.nextDouble()))
.setWrappedFloat(FloatValue.of(RANDOM.nextFloat()))
.setWrappedBoolean(BoolValue.of(RANDOM.nextBoolean()))
- .setWrappedBytes(BytesValue.of(ByteString.copyFrom(getUTF8Bytes(UUID.randomUUID().toString()))))
+ .setWrappedBytes(BytesValue.of(ByteString.copyFrom(UUID.randomUUID().toString().getBytes())))
Review Comment:
Bad merge on my part, fixing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1594942440
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -181,7 +218,7 @@ private static List<Sample> createSampleMessages(int count) {
.setWrappedDouble(DoubleValue.of(RANDOM.nextDouble()))
.setWrappedFloat(FloatValue.of(RANDOM.nextFloat()))
.setWrappedBoolean(BoolValue.of(RANDOM.nextBoolean()))
- .setWrappedBytes(BytesValue.of(ByteString.copyFrom(getUTF8Bytes(UUID.randomUUID().toString()))))
+ .setWrappedBytes(BytesValue.of(ByteString.copyFrom(UUID.randomUUID().toString().getBytes())))
Review Comment:
why removing the `getUTF8Bytes`?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -198,22 +235,48 @@ private static Nested generateRandomNestedMessage() {
@Override
void sendMessagesToKafka(String topic, int count, int numPartitions) {
List<Sample> messages = createSampleMessages(count);
- try (Producer<String, byte[]> producer = new KafkaProducer<>(getProducerProperties())) {
+ try (Producer<String, byte[]> producer = new KafkaProducer<>(getProducerProperties(false))) {
for (int i = 0; i < messages.size(); i++) {
// use consistent keys to get even spread over partitions for test expectations
producer.send(new ProducerRecord<>(topic, Integer.toString(i % numPartitions), messages.get(i).toByteArray()));
}
}
}
- private Properties getProducerProperties() {
+ private void sendMessagesToKafkaWithConfluentSerializer(String topic, int numPartitions, List<Sample> messages) {
+ try (Producer<String, Message> producer = new KafkaProducer<>(getProducerProperties(true))) {
+ for (int i = 0; i < messages.size(); i++) {
+ // use consistent keys to get even spread over partitions for test expectations
+ producer.send(new ProducerRecord<>(topic, Integer.toString(i % numPartitions), messages.get(i)));
+ }
+ }
+ }
+
+ private Properties getProducerProperties(boolean useConfluentProtobufSerializer) {
Properties props = new Properties();
props.put("bootstrap.servers", testUtils.brokerAddress());
- props.put("value.serializer", ByteArraySerializer.class.getName());
- // Key serializer is required.
+ if (useConfluentProtobufSerializer) {
+ props.put("value.serializer",
+ "io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer");
+ props.put("value.deserializer",
+ "io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer");
Review Comment:
Instead of hard-coding, how about `KafkaProtobufSerializer.class.getName()`? Is it that we don't want to load the imports of external class?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deser/KafkaAvroSchemaDeserializer.java:
##########
@@ -70,13 +69,12 @@ public void configure(Map<String, ?> configs, boolean isKey) {
*/
@Override
protected Object deserialize(
- boolean includeSchemaAndVersion,
String topic,
Boolean isKey,
byte[] payload,
Schema readerSchema)
throws SerializationException {
- return super.deserialize(includeSchemaAndVersion, topic, isKey, payload, sourceSchema);
+ return super.deserialize(topic, isKey, payload, sourceSchema);
Review Comment:
Did we ever not include schema and version?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -402,7 +438,21 @@ private static Object convertObject(Schema schema, Object value) {
if (value instanceof Message) {
// check if this is a Timestamp
if (LogicalTypes.timestampMicros().equals(schema.getLogicalType())) {
- return Timestamps.toMicros((Timestamp) value);
+ if (value instanceof Timestamp) {
+ return Timestamps.toMicros((Timestamp) value);
+ } else if (value instanceof DynamicMessage) {
+ Timestamp.Builder builder = Timestamp.newBuilder();
+ ((DynamicMessage) value).getAllFields().forEach((fieldDescriptor, fieldValue) -> {
+ if (fieldDescriptor.getFullName().equals("google.protobuf.Timestamp.seconds")) {
+ builder.setSeconds((Long) fieldValue);
+ } else if (fieldDescriptor.getFullName().equals("google.protobuf.Timestamp.nanos")) {
+ builder.setNanos((Integer) fieldValue);
+ }
Review Comment:
Should it be other way around i.e. for Timestamp.seconds, cast to Integer and Timestamp.nanos to Long?
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -133,6 +133,7 @@
<include>io.confluent:common-config</include>
<include>io.confluent:common-utils</include>
<include>io.confluent:kafka-schema-registry-client</include>
+ <include>io.confluent:kafka-protobuf-serializer</include>
Review Comment:
Should we also include this in `hudi-integ-test-bundle`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2073879258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ce1316840852fa8e21363100f6ce695a5ecf0a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2070893108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c250cc04340a016a04878a7647d4b27a608e7374 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105876784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
}, {
"hash" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23854",
"triggerID" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36cecc155c8cf9fbec29f895eea6111150eacf0b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23854)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105599191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2075688373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b49276121b062b285e1004bc72d779a54540d4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11070:
URL: https://github.com/apache/hudi/pull/11070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1597357603
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/ProtoKafkaSource.java:
##########
@@ -63,11 +67,18 @@ public ProtoKafkaSource(TypedProperties props, JavaSparkContext sparkContext, Sp
public ProtoKafkaSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession, HoodieIngestionMetrics metrics, StreamContext streamContext) {
super(properties, sparkContext, sparkSession, SourceType.PROTO, metrics,
new DefaultStreamContext(UtilHelpers.getSchemaProviderForKafkaSource(streamContext.getSchemaProvider(), properties, sparkContext), streamContext.getSourceProfileSupplier()));
- checkRequiredConfigProperties(props, Collections.singletonList(
- ProtoClassBasedSchemaProviderConfig.PROTO_SCHEMA_CLASS_NAME));
- props.put(NATIVE_KAFKA_KEY_DESERIALIZER_PROP, StringDeserializer.class);
- props.put(NATIVE_KAFKA_VALUE_DESERIALIZER_PROP, ByteArrayDeserializer.class);
- className = getStringWithAltKeys(props, ProtoClassBasedSchemaProviderConfig.PROTO_SCHEMA_CLASS_NAME);
+ this.deserializerName = ConfigUtils.getStringWithAltKeys(props, KafkaSourceConfig.KAFKA_PROTO_VALUE_DESERIALIZER_CLASS, true);
+ if (!deserializerName.equals(ByteArrayDeserializer.class.getName()) && !deserializerName.equals(KafkaProtobufDeserializer.class.getName())) {
+ throw new HoodieReadFromSourceException("Only ByteArrayDeserializer and KafkaProtobufDeserializer are supported for ProtoKafkaSource");
+ }
+ if (deserializerName.equals(ByteArrayDeserializer.class.getName())) {
+ checkRequiredConfigProperties(props, Collections.singletonList(ProtoClassBasedSchemaProviderConfig.PROTO_SCHEMA_CLASS_NAME));
+ className = getStringWithAltKeys(props, ProtoClassBasedSchemaProviderConfig.PROTO_SCHEMA_CLASS_NAME);
+ } else {
+ className = null;
Review Comment:
Avoid using `null`
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -64,21 +69,24 @@
import java.util.stream.IntStream;
import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
+import static org.apache.hudi.utilities.config.KafkaSourceConfig.KAFKA_PROTO_VALUE_DESERIALIZER_CLASS;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* Tests against {@link ProtoKafkaSource}.
*/
public class TestProtoKafkaSource extends BaseTestKafkaSource {
+ private static final JsonFormat.Printer PRINTER = JsonFormat.printer().omittingInsignificantWhitespace();
private static final Random RANDOM = new Random();
+ private static final String MOCK_REGISTRY_URL = "mock://127.0.0.1:8081";
protected TypedProperties createPropsForKafkaSource(String topic, Long maxEventsToReadFromKafkaSource, String resetStrategy) {
TypedProperties props = new TypedProperties();
- props.setProperty("hoodie.streamer.source.kafka.topic", topic);
+ props.setProperty("hoodie.deltastreamer.source.kafka.topic", topic);
Review Comment:
nit: should not be changed.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -158,7 +187,7 @@ private static List<Sample> createSampleMessages(int count) {
.setPrimitiveFixedSignedLong(RANDOM.nextLong())
.setPrimitiveBoolean(RANDOM.nextBoolean())
.setPrimitiveString(UUID.randomUUID().toString())
- .setPrimitiveBytes(ByteString.copyFrom(getUTF8Bytes(UUID.randomUUID().toString())));
+ .setPrimitiveBytes(ByteString.copyFrom(UUID.randomUUID().toString().getBytes()));
Review Comment:
similar here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2075578861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ce1316840852fa8e21363100f6ce695a5ecf0a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435)
* 04b49276121b062b285e1004bc72d779a54540d4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1595514742
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -133,6 +133,7 @@
<include>io.confluent:common-config</include>
<include>io.confluent:common-utils</include>
<include>io.confluent:kafka-schema-registry-client</include>
+ <include>io.confluent:kafka-protobuf-serializer</include>
Review Comment:
I am not sure. What is the criteria for including it in the integ test bundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2102896644
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7ef574e9785174f786bb53c3c8c84cf6514da5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776)
* 2d916b8da0cc796746409ce0512f9b80bc4d2205 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788)
* 4e15df959494651d74e92f6a998d3310bfc91247 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105577215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e15df959494651d74e92f6a998d3310bfc91247 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790)
* 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2073722347
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c250cc04340a016a04878a7647d4b27a608e7374 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403)
* 1ce1316840852fa8e21363100f6ce695a5ecf0a7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1595520788
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deser/KafkaAvroSchemaDeserializer.java:
##########
@@ -70,13 +69,12 @@ public void configure(Map<String, ?> configs, boolean isKey) {
*/
@Override
protected Object deserialize(
- boolean includeSchemaAndVersion,
String topic,
Boolean isKey,
byte[] payload,
Schema readerSchema)
throws SerializationException {
- return super.deserialize(includeSchemaAndVersion, topic, isKey, payload, sourceSchema);
+ return super.deserialize(topic, isKey, payload, sourceSchema);
Review Comment:
This option is no longer available in the confluent library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1597356380
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -133,6 +133,7 @@
<include>io.confluent:common-config</include>
<include>io.confluent:common-utils</include>
<include>io.confluent:kafka-schema-registry-client</include>
+ <include>io.confluent:kafka-protobuf-serializer</include>
Review Comment:
I think if we're not using it in the integ tests, we should not add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2101970113
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7ef574e9785174f786bb53c3c8c84cf6514da5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2101915409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b49276121b062b285e1004bc72d779a54540d4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456)
* d7ef574e9785174f786bb53c3c8c84cf6514da5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2101920386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b49276121b062b285e1004bc72d779a54540d4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456)
* d7ef574e9785174f786bb53c3c8c84cf6514da5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1595525900
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestProtoKafkaSource.java:
##########
@@ -198,22 +235,48 @@ private static Nested generateRandomNestedMessage() {
@Override
void sendMessagesToKafka(String topic, int count, int numPartitions) {
List<Sample> messages = createSampleMessages(count);
- try (Producer<String, byte[]> producer = new KafkaProducer<>(getProducerProperties())) {
+ try (Producer<String, byte[]> producer = new KafkaProducer<>(getProducerProperties(false))) {
for (int i = 0; i < messages.size(); i++) {
// use consistent keys to get even spread over partitions for test expectations
producer.send(new ProducerRecord<>(topic, Integer.toString(i % numPartitions), messages.get(i).toByteArray()));
}
}
}
- private Properties getProducerProperties() {
+ private void sendMessagesToKafkaWithConfluentSerializer(String topic, int numPartitions, List<Sample> messages) {
+ try (Producer<String, Message> producer = new KafkaProducer<>(getProducerProperties(true))) {
+ for (int i = 0; i < messages.size(); i++) {
+ // use consistent keys to get even spread over partitions for test expectations
+ producer.send(new ProducerRecord<>(topic, Integer.toString(i % numPartitions), messages.get(i)));
+ }
+ }
+ }
+
+ private Properties getProducerProperties(boolean useConfluentProtobufSerializer) {
Properties props = new Properties();
props.put("bootstrap.servers", testUtils.brokerAddress());
- props.put("value.serializer", ByteArraySerializer.class.getName());
- // Key serializer is required.
+ if (useConfluentProtobufSerializer) {
+ props.put("value.serializer",
+ "io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer");
+ props.put("value.deserializer",
+ "io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer");
Review Comment:
At the time, my intellij imports were not working. Updating this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #11070:
URL: https://github.com/apache/hudi/pull/11070#discussion_r1595526953
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ProtoConversionUtil.java:
##########
@@ -402,7 +438,21 @@ private static Object convertObject(Schema schema, Object value) {
if (value instanceof Message) {
// check if this is a Timestamp
if (LogicalTypes.timestampMicros().equals(schema.getLogicalType())) {
- return Timestamps.toMicros((Timestamp) value);
+ if (value instanceof Timestamp) {
+ return Timestamps.toMicros((Timestamp) value);
+ } else if (value instanceof DynamicMessage) {
+ Timestamp.Builder builder = Timestamp.newBuilder();
+ ((DynamicMessage) value).getAllFields().forEach((fieldDescriptor, fieldValue) -> {
+ if (fieldDescriptor.getFullName().equals("google.protobuf.Timestamp.seconds")) {
+ builder.setSeconds((Long) fieldValue);
+ } else if (fieldDescriptor.getFullName().equals("google.protobuf.Timestamp.nanos")) {
+ builder.setNanos((Integer) fieldValue);
+ }
Review Comment:
Seconds is seconds since epoch and nanos is nanos within the second so the units are as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2103021639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e15df959494651d74e92f6a998d3310bfc91247 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2071045248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c250cc04340a016a04878a7647d4b27a608e7374 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2070906076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c250cc04340a016a04878a7647d4b27a608e7374 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105574435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e15df959494651d74e92f6a998d3310bfc91247 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790)
* 86265f6be7c6fbacf53ed76a7b60b2b64d484409 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105732770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23788",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
}, {
"hash" : "4e15df959494651d74e92f6a998d3310bfc91247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23790",
"triggerID" : "4e15df959494651d74e92f6a998d3310bfc91247",
"triggerType" : "PUSH"
}, {
"hash" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851",
"triggerID" : "86265f6be7c6fbacf53ed76a7b60b2b64d484409",
"triggerType" : "PUSH"
}, {
"hash" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36cecc155c8cf9fbec29f895eea6111150eacf0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23851)
* 36cecc155c8cf9fbec29f895eea6111150eacf0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11070:
URL: https://github.com/apache/hudi/pull/11070#issuecomment-2102785650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c250cc04340a016a04878a7647d4b27a608e7374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23403",
"triggerID" : "c250cc04340a016a04878a7647d4b27a608e7374",
"triggerType" : "PUSH"
}, {
"hash" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23435",
"triggerID" : "1ce1316840852fa8e21363100f6ce695a5ecf0a7",
"triggerType" : "PUSH"
}, {
"hash" : "04b49276121b062b285e1004bc72d779a54540d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23456",
"triggerID" : "04b49276121b062b285e1004bc72d779a54540d4",
"triggerType" : "PUSH"
}, {
"hash" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776",
"triggerID" : "d7ef574e9785174f786bb53c3c8c84cf6514da5d",
"triggerType" : "PUSH"
}, {
"hash" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d916b8da0cc796746409ce0512f9b80bc4d2205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7ef574e9785174f786bb53c3c8c84cf6514da5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23776)
* 2d916b8da0cc796746409ce0512f9b80bc4d2205 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org