You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/02/15 19:20:27 UTC
[GitHub] [hudi] jonvex opened a new pull request, #7971: add support for kafka offsets
jonvex opened a new pull request, #7971:
URL: https://github.com/apache/hudi/pull/7971
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1431933577
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c96468c362b0b256093b62c5ecdcf7f64a9c7acc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1434907377
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
}, {
"hash" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9950b88e3aee7eabb8886a1543834b573ea12162 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257)
* f6a221c924464b81f5b695e8b74e9260e827f7df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1431918714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c96468c362b0b256093b62c5ecdcf7f64a9c7acc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1434916403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
}, {
"hash" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15279",
"triggerID" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9950b88e3aee7eabb8886a1543834b573ea12162 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257)
* f6a221c924464b81f5b695e8b74e9260e827f7df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1434174892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9950b88e3aee7eabb8886a1543834b573ea12162 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1432398016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c96468c362b0b256093b62c5ecdcf7f64a9c7acc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1435481621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
}, {
"hash" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15279",
"triggerID" : "f6a221c924464b81f5b695e8b74e9260e827f7df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f6a221c924464b81f5b695e8b74e9260e827f7df Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1433832914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c96468c362b0b256093b62c5ecdcf7f64a9c7acc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214)
* 9950b88e3aee7eabb8886a1543834b573ea12162 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7971:
URL: https://github.com/apache/hudi/pull/7971#issuecomment-1433846676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214",
"triggerID" : "c96468c362b0b256093b62c5ecdcf7f64a9c7acc",
"triggerType" : "PUSH"
}, {
"hash" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257",
"triggerID" : "9950b88e3aee7eabb8886a1543834b573ea12162",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c96468c362b0b256093b62c5ecdcf7f64a9c7acc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15214)
* 9950b88e3aee7eabb8886a1543834b573ea12162 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex closed pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex closed pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
URL: https://github.com/apache/hudi/pull/7971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7971:
URL: https://github.com/apache/hudi/pull/7971#discussion_r1108704365
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java:
##########
@@ -77,16 +78,33 @@ public AvroKafkaSource(TypedProperties props, JavaSparkContext sparkContext, Spa
@Override
JavaRDD<GenericRecord> toRDD(OffsetRange[] offsetRanges) {
+ JavaRDD<ConsumerRecord<Object, Object>> kafkaRDD;
if (deserializerClassName.equals(ByteArrayDeserializer.class.getName())) {
if (schemaProvider == null) {
throw new HoodieException("Please provide a valid schema provider class when use ByteArrayDeserializer!");
}
- AvroConvertor convertor = new AvroConvertor(schemaProvider.getSourceSchema());
- return KafkaUtils.<String, byte[]>createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
- LocationStrategies.PreferConsistent()).map(obj -> convertor.fromAvroBinary(obj.value()));
+ AvroConvertor convertor = getAvroConverter(true);
+ kafkaRDD = KafkaUtils.<String, byte[]>createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
+ LocationStrategies.PreferConsistent()).map(obj ->
+ new ConsumerRecord<>(obj.topic(), obj.partition(), obj.offset(),
+ obj.key(), convertor.fromAvroBinary(obj.value())));
} else {
- return KafkaUtils.createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
- LocationStrategies.PreferConsistent()).map(obj -> (GenericRecord) obj.value());
+ kafkaRDD = KafkaUtils.createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
+ LocationStrategies.PreferConsistent());
}
+ return processForKafkaOffsets(kafkaRDD);
}
+
+ protected JavaRDD<GenericRecord> processForKafkaOffsets(JavaRDD<ConsumerRecord<Object, Object>> kafkaRDD) {
Review Comment:
nit: rename to "mayBeAppendKafkaOffsets"
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java:
##########
@@ -52,13 +61,30 @@ public JsonKafkaSource(TypedProperties properties, JavaSparkContext sparkContext
@Override
JavaRDD<String> toRDD(OffsetRange[] offsetRanges) {
- JavaRDD<String> jsonStringRDD = KafkaUtils.createRDD(sparkContext,
+ JavaRDD<ConsumerRecord<Object, Object>> kafkaRDD = KafkaUtils.createRDD(sparkContext,
offsetGen.getKafkaParams(),
offsetRanges,
LocationStrategies.PreferConsistent())
- .filter(x -> !StringUtils.isNullOrEmpty((String) x.value()))
- .map(x -> x.value().toString());
- return postProcess(jsonStringRDD);
+ .filter(x -> !StringUtils.isNullOrEmpty((String) x.value()));
+
+ JavaRDD<String> result = shouldAppendKafkaOffsets() ? kafkaRDD.mapPartitions(partitionIterator -> {
Review Comment:
can we move the appending of offsets to a private method and keep this method lean (as you did for AvroKafkaSource)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaProviderWithPostProcessorDisableSource.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+
+public class SchemaProviderWithPostProcessorDisableSource extends SchemaProviderWithPostProcessor {
+ private final SchemaProvider schemaProvider;
+ private final Option<SchemaPostProcessor> schemaPostProcessor;
+ public static class Config {
+ public static final String DISABLE_POST_PROCESSOR_ON_SOURCE_SCHEMA =
Review Comment:
I mean, we should not name this class as SchemaProviderWithPostProcessor"Disable"Source.
Consider this as general schema handling where optionaly post processor could be added for source and target schema.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KafkaSource.java:
##########
@@ -76,4 +86,14 @@ public void onCommit(String lastCkptStr) {
offsetGen.commitOffsetToKafka(lastCkptStr);
}
}
+
+ protected AvroConvertor getAvroConverter(Boolean preProcessedSchema) {
+ if (schemaProvider == null) {
+ throw new HoodieException("Needed schema provider for avro deserializer");
+ }
+ if (preProcessedSchema && (schemaProvider instanceof SchemaProviderWithPostProcessorDisableSource)) {
+ return new AvroConvertor(((SchemaProviderWithPostProcessorDisableSource) schemaProvider).getSourceSchemaWithoutPostProcessor());
Review Comment:
this is what I was pointing out earlier. We should call a generic method. internally whether post processor for source is enabled/disabled should not matter to callers.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KafkaSource.java:
##########
@@ -50,6 +56,10 @@ protected KafkaSource(TypedProperties props, JavaSparkContext sparkContext, Spar
this.metrics = metrics;
}
+ protected boolean shouldAppendKafkaOffsets() {
+ return KafkaOffsetPostProcessor.class.getName().equals(props.getString(SCHEMA_POST_PROCESSOR_PROP, null));
Review Comment:
there could be multiple entries in the schema post processor config right ? should we not parse the config value and find if "KafkaOffsetPostProcessor.class.getName()" is one of them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7971:
URL: https://github.com/apache/hudi/pull/7971#discussion_r1108700794
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaProviderWithPostProcessorDisableSource.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+
+public class SchemaProviderWithPostProcessorDisableSource extends SchemaProviderWithPostProcessor {
+ private final SchemaProvider schemaProvider;
+ private final Option<SchemaPostProcessor> schemaPostProcessor;
+ public static class Config {
+ public static final String DISABLE_POST_PROCESSOR_ON_SOURCE_SCHEMA =
Review Comment:
I feel we should make this generic. Introduce this config as a generic config and apply it to any schema provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #7971: [HUDI-5808] Add Support for kaffka ofsets in jsonkafkasource and avrokafkasource
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #7971:
URL: https://github.com/apache/hudi/pull/7971#discussion_r1109060503
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KafkaSource.java:
##########
@@ -50,6 +56,10 @@ protected KafkaSource(TypedProperties props, JavaSparkContext sparkContext, Spar
this.metrics = metrics;
}
+ protected boolean shouldAppendKafkaOffsets() {
+ return KafkaOffsetPostProcessor.class.getName().equals(props.getString(SCHEMA_POST_PROCESSOR_PROP, null));
Review Comment:
I have no idea, but I updated it to do what you suggest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org