You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/02/19 22:07:18 UTC
[PR] feat:Create KinesisSource [hudi]
linliu-code opened a new pull request, #18224:
URL: https://github.com/apache/hudi/pull/18224
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3938594350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 01a34fe2172609ecab7c865f2e3052bc82e6cc9c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174)
* d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937884929
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 82ade47dcdb28b4fd8db26ff39980c5e72fca27b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172)
* 01a34fe2172609ecab7c865f2e3052bc82e6cc9c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937272853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* d09affde11485fedf2aba0f93230da54669bf307 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139)
* a517e2685c28a1101af008e73abfe04522484a17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2909637834
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
Review Comment:
Haven't figured out how this is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2909691909
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ if (minPartitions > 0 && minPartitions > shardRanges.length) {
+ int targetPartitions = (int) minPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (minPartitions={})",
+ shardRanges.length, targetPartitions, minPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+
+ // Information for checkpoint generation.
+ List<ShardFetchSummary> summaries = fetchRdd
+ .map(r -> new ShardFetchSummary(r.getShardId(), r.getLastSequenceNumber(), r.getRecords().size(),
+ r.isReachedEndOfShard()))
+ .collect();
+ lastCheckpointData = buildCheckpointFromSummaries(summaries);
+ Set<String> reached = new HashSet<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.isReachedEndOfShard()) {
+ reached.add(s.getShardId());
+ }
+ }
+ shardsReachedEnd = reached;
+ lastRecordCount = summaries.stream().mapToLong(ShardFetchSummary::getRecordCount).sum();
+
+ return recordRdd;
+ }
+
+ private static KinesisClient createKinesisClientFromConfig(KinesisReadConfig config) {
+ software.amazon.awssdk.services.kinesis.KinesisClientBuilder builder =
+ KinesisClient.builder().region(software.amazon.awssdk.regions.Region.of(config.getRegion()));
+ if (config.getEndpointUrl() != null && !config.getEndpointUrl().isEmpty()) {
+ builder = builder.endpointOverride(java.net.URI.create(config.getEndpointUrl()));
+ }
+ if (config.getAccessKey() != null && !config.getAccessKey().isEmpty()
+ && config.getSecretKey() != null && !config.getSecretKey().isEmpty()) {
+ builder = builder.credentialsProvider(
+ software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
+ software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create(
+ config.getAccessKey(), config.getSecretKey())));
+ }
+ return builder.build();
+ }
+
+ private static String recordToJsonStatic(Record record, String shardId, boolean shouldAddOffsets) {
+ String dataStr = record.data().asUtf8String();
+ // Pure empty or null records in Kinesis is not meaningful.
+ if (dataStr == null || dataStr.trim().isEmpty()) {
+ return null;
+ }
+ if (shouldAddOffsets) {
+ try {
+ ObjectNode node = (ObjectNode) OBJECT_MAPPER.readTree(dataStr);
+ node.put("_hoodie_kinesis_source_sequence_number", record.sequenceNumber());
+ node.put("_hoodie_kinesis_source_shard_id", shardId);
+ node.put("_hoodie_kinesis_source_partition_key", record.partitionKey());
+ if (record.approximateArrivalTimestamp() != null) {
+ node.put("_hoodie_kinesis_source_timestamp",
+ record.approximateArrivalTimestamp().toEpochMilli());
+ }
+ return OBJECT_MAPPER.writeValueAsString(node);
+ } catch (Exception e) {
+ return dataStr;
+ }
+ }
+ return dataStr;
+ }
+
+ private Map<String, String> buildCheckpointFromSummaries(List<ShardFetchSummary> summaries) {
+ Map<String, String> checkpoint = new HashMap<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.getLastSequenceNumber().isPresent()) {
+ checkpoint.put(s.getShardId(), s.getLastSequenceNumber().get());
+ }
+ }
+ return checkpoint;
+ }
+
+ @Override
+ protected String createCheckpointFromBatch(JavaRDD<String> batch,
+ KinesisOffsetGen.KinesisShardRange[] shardRangesRead,
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges) {
+ // Build checkpoint from ALL shards so filtered-out (fully consumed) shards are preserved.
+ // Otherwise the next run would omit them from the checkpoint and re-read from TRIM_HORIZON.
+ Map<String, String> fullCheckpoint = new HashMap<>();
+ for (KinesisOffsetGen.KinesisShardRange range : allShardRanges) {
+ String lastSeq;
+ String endSeq;
+ if (lastCheckpointData != null && lastCheckpointData.containsKey(range.getShardId())) {
+ // Shard we read from: use data from the read
+ lastSeq = lastCheckpointData.get(range.getShardId());
+ endSeq = range.getEndingSequenceNumber().orElse(null);
+ } else {
+ // Filtered shard (not read): preserve from range so next run won't re-read
+ lastSeq = range.getStartingSequenceNumber().orElse("");
+ endSeq = range.getEndingSequenceNumber().orElse(null);
+ }
+ // for test only
+ // LocalStack returns Long.MAX_VALUE for closed shards; use lastSeq as endSeq so we can detect
+ // "fully consumed" when the parent shard expires (lastSeq >= endSeq).
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && lastSeq != null && !lastSeq.isEmpty()) {
+ endSeq = lastSeq;
+ }
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2843059739
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
Review Comment:
```suggestion
.key(PREFIX + "fail.on.data.loss.enable")
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
+ .key(PREFIX + "getRecords.maxRecords")
+ .defaultValue(10000)
+ .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
Review Comment:
Is there a validation on the config?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
Review Comment:
Similar for `TRIM_HORIZON`
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
Review Comment:
Use `EARLIEST` instead of `TRIM_HORIZON` for readability?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
+ .key(PREFIX + "getRecords.maxRecords")
Review Comment:
```suggestion
.key(PREFIX + "max.records.per.request")
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
+ .key(PREFIX + "getRecords.maxRecords")
+ .defaultValue(10000)
+ .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
+
+ public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "getRecords.intervalMs")
+ .defaultValue(200L)
+ .withAlternatives(OLD_PREFIX + "getRecords.intervalMs")
+ .markAdvanced()
+ .withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
Review Comment:
Why do we need this config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2898461904
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
Review Comment:
yes. No expired shards will be returned. So these expired shards will not be included in the checkpoint anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2872033566
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
Review Comment:
yeah, we can add a configuration to control it, by default it is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328189535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
}, {
"hash" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13652",
"triggerID" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"triggerType" : "PUSH"
}, {
"hash" : "669d312a57cc716458dc8f03d8923081cabe8124",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "669d312a57cc716458dc8f03d8923081cabe8124",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* 7af0201256fc80be36c78bc2e56df221b876902a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650)
* a193e7e13b56a25526e18c3eec95cbdf3d16c79c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13652)
* 669d312a57cc716458dc8f03d8923081cabe8124 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324090979
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 44.90%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 21 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.90% -24.00%
+ Complexity 28537 15679 -12858
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 51299 -43023
- Misses 34996 57420 +22424
+ Partials 7592 5527 -2065
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1595 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324187228
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.23%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.23% -2.66%
+ Complexity 28537 23104 -5433
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75675 -18647
+ Misses 34996 31798 -3198
+ Partials 7592 6773 -819
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.92% <0.00%> (-0.56%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.99% <0.00%> (-0.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [843 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325642665
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.66%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|7|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.66% -9.24%
+ Complexity 28537 21000 -7537
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 69749 -24573
- Misses 34996 40049 +5053
+ Partials 7592 7107 -485
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.64% <0.00%> (-17.83%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <0.00%> (-0.50%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1281 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329124725
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.79%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 67.79% -1.11%
- Complexity 28537 28752 +215
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95274 +952
- Misses 34996 37463 +2467
- Partials 7592 7801 +209
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.93% <0.00%> (-1.54%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.62% <0.00%> (-0.66%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [114 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324284546
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.33%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.33% -2.56%
+ Complexity 28537 23156 -5381
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75785 -18537
+ Misses 34996 31691 -3305
+ Partials 7592 6770 -822
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.21% <0.00%> (-0.26%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.99% <0.00%> (-0.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [829 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034794974
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
Review Comment:
I think you got a point; and I have already added thecheck.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
Review Comment:
Will.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034286227
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
Review Comment:
Same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034351954
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
+ }
+
+ T batch = toBatch(shardRanges);
+ // Pass allShardRanges so filtered-out shards are preserved in the checkpoint; otherwise
+ // next run would re-read them from TRIM_HORIZON and cause duplicates
+ String checkpointStr = createCheckpointFromBatch(batch, shardRanges, allShardRanges);
+ long totalMsgs = getRecordCount(batch);
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, totalMsgs);
+
+ log.info("Read {} records from Kinesis stream {} with {} shards, checkpoint: {}",
+ totalMsgs, offsetGen.getStreamName(), shardRanges.length, checkpointStr);
Review Comment:
With caching this is OK for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930781533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129)
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* fec6001976389d10a38feda0a413fb6043f23ace Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2849642962
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
Review Comment:
```suggestion
int lengthBeforeFilter = shardRanges.length;
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
Review Comment:
Rename this to `startingStrategy` or something similar to be readable?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
Review Comment:
Could there be a case where the checkpoint needs to be set, instead of empty string, and there is no message to ingest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3945078567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 917c414dba55734c588d3c43c4b0e16e137fb211 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2898517584
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -152,6 +152,9 @@
<include>com.101tec:zkclient</include>
<include>org.apache.kafka:kafka-clients</include>
+ <!-- AWS Kinesis SDK for JsonKinesisSource -->
+ <include>software.amazon.awssdk:kinesis</include>
+
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4028505976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 8d77ec07f756f71fb3105c8a38008a995b05d793 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2909713532
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913266149
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
Review Comment:
We could. Just fyi that since the sequence number in a shard is not consecutive, there could be false alarms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034718458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 8d77ec07f756f71fb3105c8a38008a995b05d793 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517)
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2898517584
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -152,6 +152,9 @@
<include>com.101tec:zkclient</include>
<include>org.apache.kafka:kafka-clients</include>
+ <!-- AWS Kinesis SDK for JsonKinesisSource -->
+ <include>software.amazon.awssdk:kinesis</include>
+
Review Comment:
Here should be the right places, since Hudi-utilities-bundle does not include hudi-aws-bundle as dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913825112
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
Review Comment:
I tried to do it a couple of times. Let me try it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2914509826
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
Review Comment:
This should happen when `maxTotalRecords` is small; when it is large enough, like 5M, the `allRecords` size should be very closed to `maxTotalRecords`, since only the last page may contain some more records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034664303
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
Review Comment:
function `parseCheckpointValue` does that. I don't merge these two functions to make it more testable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185778912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062)
* 3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034263481
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
Review Comment:
Then could we add the particular check to closed shard only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185812417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062)
* 3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064)
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325260932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 08ae17785490db6a2b9432acbc3196517ed73845 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633)
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638)
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* db19834cf6915d0a9e1674ed36c04c2a45d61545 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327447528
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `70.08130%` with `184 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.96%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [55 Missing and 30 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 71.62% | [28 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.96% -8.93%
+ Complexity 28537 21157 -7380
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 70102 -24220
- Misses 34996 39586 +4590
+ Partials 7592 7217 -375
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.99% <0.00%> (-17.48%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.40% <0.00%> (-0.88%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <70.08%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `71.62% <71.62%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1279 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327451122
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `70.08130%` with `184 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.96%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [55 Missing and 30 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 71.62% | [28 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.96% -8.93%
+ Complexity 28537 21157 -7380
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 70102 -24220
- Misses 34996 39586 +4590
+ Partials 7592 7217 -375
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.99% <0.00%> (-17.48%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.40% <0.00%> (-0.88%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <70.08%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `71.62% <71.62%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1279 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327441715
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.00%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 55.00% -13.90%
+ Complexity 28537 19229 -9308
=============================================
Files 2480 2026 -454
Lines 136910 114421 -22489
Branches 16679 14664 -2015
=============================================
- Hits 94322 62935 -31387
- Misses 34996 44954 +9958
+ Partials 7592 6532 -1060
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.77% <0.00%> (-16.71%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.68% <0.00%> (-0.60%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1410 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326933681
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.81%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`937bbb2`](https://app.codecov.io/gh/apache/hudi/commit/937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (937bbb2). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (937bbb2) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.81% -24.09%
+ Complexity 28537 8519 -20018
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28063 -66259
+ Misses 34996 31474 -3522
+ Partials 7592 3089 -4503
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.81% <ø> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327890134
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.61%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7af0201`](https://app.codecov.io/gh/apache/hudi/commit/7af0201256fc80be36c78bc2e56df221b876902a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (7af0201). Click for more details.
>
> <details><summary>HEAD has 21 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (7af0201) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.61% -20.29%
+ Complexity 28537 16584 -11953
=============================================
Files 2480 2008 -472
Lines 136910 111999 -24911
Branches 16679 14145 -2534
=============================================
- Hits 94322 54446 -39876
- Misses 34996 51802 +16806
+ Partials 7592 5751 -1841
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.02%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.76% <0.00%> (-16.71%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.88% <0.00%> (-12.39%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1563 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324687894
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.57%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.57% -21.32%
+ Complexity 28537 16578 -11959
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 54354 -39968
- Misses 34996 54155 +19159
+ Partials 7592 5737 -1855
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.10% <0.00%> (-16.38%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325766184
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.80%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.80% -3.09%
+ Complexity 28537 23512 -5025
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76935 -17387
+ Misses 34996 32965 -2031
+ Partials 7592 7005 -587
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.57% <0.00%> (-0.90%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [862 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328386671
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.82%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`669d312`](https://app.codecov.io/gh/apache/hudi/commit/669d312a57cc716458dc8f03d8923081cabe8124?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (669d312). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (669d312) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.82% -24.07%
+ Complexity 28537 8522 -20015
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28073 -66249
+ Misses 34996 31467 -3529
+ Partials 7592 3086 -4506
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328917323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
}, {
"hash" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13652",
"triggerID" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"triggerType" : "PUSH"
}, {
"hash" : "669d312a57cc716458dc8f03d8923081cabe8124",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13655",
"triggerID" : "669d312a57cc716458dc8f03d8923081cabe8124",
"triggerType" : "PUSH"
}, {
"hash" : "6f2372e844600c92ced164a44d8ef70ed742d004",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13656",
"triggerID" : "6f2372e844600c92ced164a44d8ef70ed742d004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* 669d312a57cc716458dc8f03d8923081cabe8124 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13655)
* 6f2372e844600c92ced164a44d8ef70ed742d004 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13656)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328901467
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.93%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 53.93% -14.96%
+ Complexity 28537 12392 -16145
=============================================
Files 2480 1433 -1047
Lines 136910 71927 -64983
Branches 16679 8203 -8476
=============================================
- Hits 94322 38797 -55525
+ Misses 34996 29651 -5345
+ Partials 7592 3479 -4113
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1829 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325750257
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.72%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.72% -3.17%
+ Complexity 28537 23468 -5069
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76835 -17487
+ Misses 34996 33084 -1912
+ Partials 7592 6986 -606
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.42% <0.00%> (-1.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.61% <0.00%> (-0.67%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [865 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325725429
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.60%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.60% -3.29%
+ Complexity 28537 23391 -5146
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76696 -17626
+ Misses 34996 33231 -1765
+ Partials 7592 6978 -614
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.23% <0.00%> (-1.25%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.54% <0.00%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [874 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326534451
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* 44f0050b30e808c35359269a6da7aa6bad6f926e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642)
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324126216
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.87%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 13 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|7|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.87% -9.03%
+ Complexity 28537 21194 -7343
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 68401 -25921
- Misses 34996 39605 +4609
+ Partials 7592 6240 -1352
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.17% <0.00%> (-4.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.61% <0.00%> (-0.67%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1107 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324071336
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.31%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|1|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 46.31% -22.59%
+ Complexity 28537 15123 -13414
=============================================
Files 2480 1961 -519
Lines 136910 106734 -30176
Branches 16679 12971 -3708
=============================================
- Hits 94322 49432 -44890
- Misses 34996 52214 +17218
+ Partials 7592 5088 -2504
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.69% <0.00%> (-12.59%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1653 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937247622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* d09affde11485fedf2aba0f93230da54669bf307 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139)
* a517e2685c28a1101af008e73abfe04522484a17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937802550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a517e2685c28a1101af008e73abfe04522484a17 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166)
* 82ade47dcdb28b4fd8db26ff39980c5e72fca27b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172)
* 01a34fe2172609ecab7c865f2e3052bc82e6cc9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2849444224
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
Review Comment:
We should also integrate `sourceProfileSupplier` (`SourceProfileSupplier`) or make sure to track that as as a follow-up.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
Review Comment:
We should avoid storing the read records in a list which can significantly increase the memory usage. We should use the iterator pattern to reduce the memory pressure on the executor side.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
+
+ // Get here when
+ // 1. reach the max total record limit
+ // 2. or reach the end of the closed shard
+ // 3. or the open shard does not have any more records in last iteration.
+ log.debug("Read {} records from shard {} in {} requests", allRecords.size(), range.getShardId(), requestCount);
+ return new ShardReadResult(allRecords, Option.ofNullable(lastSequenceNumber), shardIterator == null);
+ }
+
+ private static String getShardIterator(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition) {
+ GetShardIteratorRequest.Builder builder = GetShardIteratorRequest.builder()
+ .streamName(streamName)
+ .shardId(range.getShardId());
+
+ if (range.getStartingSequenceNumber().isPresent()) {
+ builder.shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER);
+ builder.startingSequenceNumber(range.getStartingSequenceNumber().get());
+ } else {
+ // EARLIEST is normalized to TRIM_HORIZON in constructor
+ builder.shardIteratorType(defaultPosition == KinesisSourceConfig.KinesisStartingPosition.TRIM_HORIZON
+ ? ShardIteratorType.TRIM_HORIZON : ShardIteratorType.LATEST);
Review Comment:
`KinesisSourceConfig.KinesisStartingPosition.EARLIEST` should also map to `ShardIteratorType.TRIM_HORIZON`.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
+
+ // Get here when
+ // 1. reach the max total record limit
+ // 2. or reach the end of the closed shard
+ // 3. or the open shard does not have any more records in last iteration.
+ log.debug("Read {} records from shard {} in {} requests", allRecords.size(), range.getShardId(), requestCount);
+ return new ShardReadResult(allRecords, Option.ofNullable(lastSequenceNumber), shardIterator == null);
+ }
+
+ private static String getShardIterator(KinesisClient client, String streamName,
Review Comment:
```suggestion
private static String getCurrentCursor(KinesisClient client, String streamName,
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
Review Comment:
```suggestion
String currentCursor;
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
Review Comment:
Could we remove this as it does self rate-limiting? Instead, we should let the SDK to do the retries after rate limiting.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
Review Comment:
This assumes that the returned records are sorted based on the sequence number. Is that guaranteed?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
Review Comment:
Could we wrap the logic here into a closable iterator that can be directly used by the executor in `JsonKinesisSource#toBatch` without accumulating records in memory and then returning the iterator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034083494
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisDeaggregator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * De-aggregates KPL (Kinesis Producer Library) aggregated records into individual user records.
+ * Non-aggregated records are returned unchanged.
+ */
+public final class KinesisDeaggregator {
+
+ private KinesisDeaggregator() {
+ }
+
+ /**
+ * De-aggregate SDK v2 Kinesis records. Aggregated records (from KPL) are split into user records.
+ * Non-aggregated records pass through unchanged.
+ */
+ public static List<Record> deaggregate(List<Record> records) {
+ if (records == null || records.isEmpty()) {
+ return new ArrayList<>();
+ }
+ List<com.amazonaws.services.kinesis.model.Record> v1Records = new ArrayList<>(records.size());
+ for (Record r : records) {
+ v1Records.add(toV1Record(r));
+ }
+ List<UserRecord> userRecords = UserRecord.deaggregate(v1Records);
+ List<Record> result = new ArrayList<>(userRecords.size());
+ for (UserRecord ur : userRecords) {
+ result.add(toV2Record(ur));
+ }
Review Comment:
Got it. Does this incur additional performance overhead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034639568
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034693684
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
Review Comment:
Added into a ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034705140
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -152,6 +152,9 @@
<include>com.101tec:zkclient</include>
<include>org.apache.kafka:kafka-clients</include>
+ <!-- AWS Kinesis SDK for JsonKinesisSource -->
+ <include>software.amazon.awssdk:kinesis</include>
+
Review Comment:
moved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034648100
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
.markAdvanced()
.withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
.key(PREFIX + "enable.deaggregation")
.defaultValue(true)
- .withAlternatives(OLD_PREFIX + "enable.deaggregation")
.markAdvanced()
.withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
- .key(PREFIX + "enable.failOnDataLoss")
+ .key(PREFIX + "enable.fail.on.data.loss")
.defaultValue(false)
- .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
.markAdvanced()
- .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
.key(PREFIX + "starting.position")
.defaultValue("LATEST")
- .withAlternatives(OLD_PREFIX + "starting.position")
.markAdvanced()
- .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
- public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
- .key(PREFIX + "getRecords.maxRecords")
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
.defaultValue(10000)
- .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
.markAdvanced()
.withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
- .key(PREFIX + "getRecords.intervalMs")
+ .key(PREFIX + "get_records.interval.ms")
Review Comment:
This is to match the function name: getRecords, which I suppose to be more intuitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324156256
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 64.93%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 64.93% -3.97%
+ Complexity 28537 22631 -5906
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 74183 -20139
+ Misses 34996 33288 -1708
+ Partials 7592 6775 -817
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.17% <0.00%> (-4.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <0.00%> (-0.45%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [970 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327382762
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.73%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 23 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 45.73% -23.16%
+ Complexity 28537 15648 -12889
=============================================
Files 2480 2009 -471
Lines 136910 111980 -24930
Branches 16679 14122 -2557
=============================================
- Hits 94322 51214 -43108
- Misses 34996 55223 +20227
+ Partials 7592 5543 -2049
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.82% <0.00%> (-12.46%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1634 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327535808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* cb751bb9a7be2db62364d07202a29ca1d17f863b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648)
* 7af0201256fc80be36c78bc2e56df221b876902a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329248370
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.03%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.03% -0.86%
- Complexity 28537 28893 +356
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95618 +1296
- Misses 34996 37073 +2077
- Partials 7592 7847 +255
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.58% <0.00%> (-0.89%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [84 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326846756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* 44f0050b30e808c35359269a6da7aa6bad6f926e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642)
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* cbaa9d827092f69c83d752739372f57c1c5a3250 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645)
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326821183
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.93%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cbaa9d8`](https://app.codecov.io/gh/apache/hudi/commit/cbaa9d827092f69c83d752739372f57c1c5a3250?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cbaa9d8). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cbaa9d8) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 53.93% -14.96%
+ Complexity 28537 12394 -16143
=============================================
Files 2480 1433 -1047
Lines 136910 71927 -64983
Branches 16679 8203 -8476
=============================================
- Hits 94322 38796 -55526
+ Misses 34996 29653 -5343
+ Partials 7592 3478 -4114
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1830 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327078545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* cbaa9d827092f69c83d752739372f57c1c5a3250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645)
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* cb751bb9a7be2db62364d07202a29ca1d17f863b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327152587
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.83%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.83% -24.06%
+ Complexity 28537 8524 -20013
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28078 -66244
+ Misses 34996 31463 -3533
+ Partials 7592 3085 -4507
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329057932
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.55%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 12 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|7|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.55% -9.35%
+ Complexity 28537 21068 -7469
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15204 -1475
============================================
- Hits 94322 69617 -24705
- Misses 34996 40139 +5143
+ Partials 7592 7149 -443
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.59% <0.00%> (-17.88%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.32% <0.00%> (-0.96%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1284 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329334757
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.03%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.03% -0.87%
- Complexity 28537 28893 +356
============================================
Files 2480 2518 +38
Lines 136910 140552 +3642
Branches 16679 17422 +743
============================================
+ Hits 94322 95618 +1296
- Misses 34996 37087 +2091
- Partials 7592 7847 +255
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.61% <0.00%> (-0.87%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [85 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4035239345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9af3386e49cb4c09581bce9632b306b2b4ced5de Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554)
* 5aaba50d2b78079d344e8d39c6baf0e76da9f5ae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2914644362
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
+
+ // Get here when
+ // 1. reach the max total record limit
+ // 2. or reach the end of the closed shard
+ // 3. or the open shard does not have any more records in last iteration.
+ log.debug("Read {} records from shard {} in {} requests", allRecords.size(), range.getShardId(), requestCount);
+ return new ShardReadResult(allRecords, Option.ofNullable(lastSequenceNumber), shardIterator == null);
+ }
+
+ private static String getShardIterator(KinesisClient client, String streamName,
Review Comment:
ShardIterator is from Kinesis library; hopefully it does not bring much confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4027994584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 759d3cdbf5684af44f12fb2ec7ef40c38760ab23 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483)
* f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325342791
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.82%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.82% -24.07%
+ Complexity 28537 8522 -20015
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28075 -66247
+ Misses 34996 31465 -3531
+ Partials 7592 3086 -4506
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326357811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* db19834cf6915d0a9e1674ed36c04c2a45d61545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640)
* 44f0050b30e808c35359269a6da7aa6bad6f926e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642)
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4323990352
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.84%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.84% -24.06%
+ Complexity 28537 8526 -20011
=============================================
Files 2480 1201 -1279
Lines 136910 62608 -74302
Branches 16679 6778 -9901
=============================================
- Hits 94322 28075 -66247
+ Misses 34996 31450 -3546
+ Partials 7592 3083 -4509
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4323963935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065)
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 282c1bea218876530b319f14dedd7a75a3fff469 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324730346
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 54.23%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 54.23% -14.66%
+ Complexity 28537 18633 -9904
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 61960 -32362
- Misses 34996 45899 +10903
+ Partials 7592 6387 -1205
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.99% <0.00%> (-17.49%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1413 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325619344
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 54.01%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 54.01% -14.88%
+ Complexity 28537 19265 -9272
=============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
=============================================
- Hits 94322 63151 -31171
- Misses 34996 47139 +12143
+ Partials 7592 6615 -977
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.03% <0.00%> (-17.44%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.45% <0.00%> (-0.83%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1420 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325732970
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.60%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.60% -3.29%
+ Complexity 28537 23391 -5146
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76697 -17625
+ Misses 34996 33231 -1765
+ Partials 7592 6977 -615
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.23% <0.00%> (-1.25%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.54% <0.00%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [873 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2871996476
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
Review Comment:
Added a warning message for it since null string could be added into Kinesis for some cases, like separation, water marking, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034266441
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
Review Comment:
Discussed that due to how Kinesis works, reading the same shard from multiple Spark executors through getRecord calls may not increase throughput as this API has per-shard throughput limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034782294
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
Review Comment:
I think so; otherwise, it cannot guarantee the records consumed in the same shard will preserve the order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185861001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062)
* 3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064)
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325576041
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.22%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 22 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|2|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.22% -20.68%
+ Complexity 28537 16415 -12122
=============================================
Files 2480 2008 -472
Lines 136910 112038 -24872
Branches 16679 14152 -2527
=============================================
- Hits 94322 54026 -40296
- Misses 34996 52343 +17347
+ Partials 7592 5669 -1923
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.80% <0.00%> (-16.67%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.86% <0.00%> (-12.42%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1590 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325090909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 08ae17785490db6a2b9432acbc3196517ed73845 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #18224:
URL: https://github.com/apache/hudi/pull/18224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329143124
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.01%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.01% -0.88%
- Complexity 28537 28884 +347
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95589 +1267
- Misses 34996 37109 +2113
- Partials 7592 7840 +248
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.38% <0.00%> (-1.10%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [93 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2850126063
##########
hudi-utilities/pom.xml:
##########
@@ -533,6 +538,24 @@
<artifactId>sqs</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>kinesis</artifactId>
+ <version>${aws.sdk.version}</version>
+ </dependency>
+ <!-- KPL de-aggregation: extracts user records from Kinesis Producer Library aggregated records -->
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>amazon-kinesis-deaggregator</artifactId>
+ <version>1.0.3</version>
+ </dependency>
+ <!-- KPL aggregation: for creating aggregated records in tests -->
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>amazon-kinesis-aggregator</artifactId>
+ <version>1.0.3</version>
Review Comment:
Add `aws.kinesis.version` to represent the version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2843129964
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisDeaggregator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * De-aggregates KPL (Kinesis Producer Library) aggregated records into individual user records.
+ * Non-aggregated records are returned unchanged.
+ */
+public final class KinesisDeaggregator {
+
+ private KinesisDeaggregator() {
+ }
+
+ /**
+ * De-aggregate SDK v2 Kinesis records. Aggregated records (from KPL) are split into user records.
+ * Non-aggregated records pass through unchanged.
+ */
+ public static List<Record> deaggregate(List<Record> records) {
+ if (records == null || records.isEmpty()) {
+ return new ArrayList<>();
+ }
+ List<com.amazonaws.services.kinesis.model.Record> v1Records = new ArrayList<>(records.size());
+ for (Record r : records) {
+ v1Records.add(toV1Record(r));
+ }
+ List<UserRecord> userRecords = UserRecord.deaggregate(v1Records);
+ List<Record> result = new ArrayList<>(userRecords.size());
+ for (UserRecord ur : userRecords) {
+ result.add(toV2Record(ur));
+ }
Review Comment:
Why is the conversion between V1 and V2 needed? Does the AWS SDK provide an API to do this?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
Review Comment:
Avoid this test-only code in the production class.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
Review Comment:
Is `KINESIS_REGION` required?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
Review Comment:
Merge these two together to return `Pair<String, Option<String>>` of last seq and optional end seq: `getLastAndEndSeqFromCheckpoint`
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
Review Comment:
Is this for paginated responses? Is there an API handling this for reuse, instead of hand-crafting the logic again?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
Review Comment:
Should we also catch other exceptions? If not, where are those caught or thrown in the upper caller chain?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
Review Comment:
Could `HoodieConfigAWSCredentialsProvider` be reused?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
Review Comment:
```suggestion
public static final ConfigProperty<KinesisStartingPosition> KINESIS_STARTING_POSITION = ConfigProperty
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
Review Comment:
Remove this test-only code.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
Review Comment:
It looks like this diverges from Kafka source's behavior of picking `sourceLimit` if it is not `Long.MAX_VALUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2872747768
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ if (minPartitions > 0 && minPartitions > shardRanges.length) {
+ int targetPartitions = (int) minPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (minPartitions={})",
+ shardRanges.length, targetPartitions, minPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+
+ // Information for checkpoint generation.
+ List<ShardFetchSummary> summaries = fetchRdd
+ .map(r -> new ShardFetchSummary(r.getShardId(), r.getLastSequenceNumber(), r.getRecords().size(),
+ r.isReachedEndOfShard()))
+ .collect();
+ lastCheckpointData = buildCheckpointFromSummaries(summaries);
+ Set<String> reached = new HashSet<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.isReachedEndOfShard()) {
+ reached.add(s.getShardId());
+ }
+ }
+ shardsReachedEnd = reached;
+ lastRecordCount = summaries.stream().mapToLong(ShardFetchSummary::getRecordCount).sum();
+
+ return recordRdd;
+ }
+
+ private static KinesisClient createKinesisClientFromConfig(KinesisReadConfig config) {
+ software.amazon.awssdk.services.kinesis.KinesisClientBuilder builder =
+ KinesisClient.builder().region(software.amazon.awssdk.regions.Region.of(config.getRegion()));
+ if (config.getEndpointUrl() != null && !config.getEndpointUrl().isEmpty()) {
+ builder = builder.endpointOverride(java.net.URI.create(config.getEndpointUrl()));
+ }
+ if (config.getAccessKey() != null && !config.getAccessKey().isEmpty()
+ && config.getSecretKey() != null && !config.getSecretKey().isEmpty()) {
+ builder = builder.credentialsProvider(
+ software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
+ software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create(
+ config.getAccessKey(), config.getSecretKey())));
+ }
+ return builder.build();
+ }
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913822243
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
Review Comment:
The SDK does not provide the loop logic OOM since it is designed for mainly streaming scenarioes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2912857114
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2914638180
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
Review Comment:
> Could there be a case where the checkpoint needs to be set, instead of empty string, and there is no message to ingest?
AFAIK, there are not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327275170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* cb751bb9a7be2db62364d07202a29ca1d17f863b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327505882
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.87805%` with `93 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.20%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 78.28% | [23 Missing and 20 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 82.43% | [14 Missing and 12 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 84.55% | [15 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 92.30% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.20% -2.69%
+ Complexity 28537 28077 -460
============================================
Files 2480 2518 +38
Lines 136910 140524 +3614
Branches 16679 17415 +736
============================================
- Hits 94322 93039 -1283
- Misses 34996 39670 +4674
- Partials 7592 7815 +223
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.31% <59.02%> (-0.11%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.57% <0.00%> (-9.90%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.53% <0.00%> (-0.75%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <70.08%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `92.30% <92.30%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `84.55% <84.55%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `82.43% <82.43%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `78.28% <78.28%> (ø)` | |
... and [303 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327502988
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.87805%` with `93 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.20%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 78.28% | [23 Missing and 20 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 82.43% | [14 Missing and 12 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 84.55% | [15 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 92.30% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.20% -2.69%
+ Complexity 28537 28077 -460
============================================
Files 2480 2518 +38
Lines 136910 140524 +3614
Branches 16679 17415 +736
============================================
- Hits 94322 93039 -1283
- Misses 34996 39670 +4674
- Partials 7592 7815 +223
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.31% <59.02%> (-0.11%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.57% <0.00%> (-9.90%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.53% <0.00%> (-0.75%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <70.08%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `92.30% <92.30%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `84.55% <84.55%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `82.43% <82.43%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `78.28% <78.28%> (ø)` | |
... and [303 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327317459
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.93%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 53.93% -14.96%
+ Complexity 28537 12392 -16145
=============================================
Files 2480 1433 -1047
Lines 136910 71927 -64983
Branches 16679 8203 -8476
=============================================
- Hits 94322 38794 -55528
+ Misses 34996 29653 -5343
+ Partials 7592 3480 -4112
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1830 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329011643
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.62%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 21 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.62% -20.27%
+ Complexity 28537 16576 -11961
=============================================
Files 2480 2009 -471
Lines 136910 112020 -24890
Branches 16679 14154 -2525
=============================================
- Hits 94322 54471 -39851
- Misses 34996 51780 +16784
+ Partials 7592 5769 -1823
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.74% <0.00%> (-16.73%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.92% <0.00%> (-12.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1563 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4323948926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065)
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326931666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* cbaa9d827092f69c83d752739372f57c1c5a3250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645)
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325758033
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.76%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.76% -3.13%
+ Complexity 28537 23492 -5045
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76885 -17437
+ Misses 34996 33027 -1969
+ Partials 7592 6993 -599
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.42% <0.00%> (-1.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [861 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324831177
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.44%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.44% -2.45%
+ Complexity 28537 23178 -5359
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 75910 -18412
+ Misses 34996 31574 -3422
+ Partials 7592 6762 -830
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.59% <0.00%> (-0.88%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [853 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324862825
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.61%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.61% -2.28%
+ Complexity 28537 23251 -5286
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76106 -18216
+ Misses 34996 31359 -3637
+ Partials 7592 6781 -811
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.30%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [820 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324881151
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.62%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.62% -2.28%
+ Complexity 28537 23252 -5285
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76111 -18211
+ Misses 34996 31356 -3640
+ Partials 7592 6779 -813
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [819 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937433199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a517e2685c28a1101af008e73abfe04522484a17 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3946561741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 917c414dba55734c588d3c43c4b0e16e137fb211 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216)
* a28e9e73447bf984536783a76949fd1f98f531dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3946898831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a28e9e73447bf984536783a76949fd1f98f531dc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324857621
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.61%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.61% -2.28%
+ Complexity 28537 23251 -5286
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76106 -18216
+ Misses 34996 31359 -3637
+ Partials 7592 6781 -811
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.30%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [820 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4323979771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065)
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 282c1bea218876530b319f14dedd7a75a3fff469 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2898225089
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ if (minPartitions > 0 && minPartitions > shardRanges.length) {
+ int targetPartitions = (int) minPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (minPartitions={})",
+ shardRanges.length, targetPartitions, minPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+
+ // Information for checkpoint generation.
+ List<ShardFetchSummary> summaries = fetchRdd
+ .map(r -> new ShardFetchSummary(r.getShardId(), r.getLastSequenceNumber(), r.getRecords().size(),
+ r.isReachedEndOfShard()))
+ .collect();
+ lastCheckpointData = buildCheckpointFromSummaries(summaries);
+ Set<String> reached = new HashSet<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.isReachedEndOfShard()) {
+ reached.add(s.getShardId());
+ }
+ }
+ shardsReachedEnd = reached;
+ lastRecordCount = summaries.stream().mapToLong(ShardFetchSummary::getRecordCount).sum();
+
+ return recordRdd;
+ }
+
+ private static KinesisClient createKinesisClientFromConfig(KinesisReadConfig config) {
+ software.amazon.awssdk.services.kinesis.KinesisClientBuilder builder =
+ KinesisClient.builder().region(software.amazon.awssdk.regions.Region.of(config.getRegion()));
+ if (config.getEndpointUrl() != null && !config.getEndpointUrl().isEmpty()) {
+ builder = builder.endpointOverride(java.net.URI.create(config.getEndpointUrl()));
+ }
+ if (config.getAccessKey() != null && !config.getAccessKey().isEmpty()
+ && config.getSecretKey() != null && !config.getSecretKey().isEmpty()) {
+ builder = builder.credentialsProvider(
+ software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
+ software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create(
+ config.getAccessKey(), config.getSecretKey())));
+ }
+ return builder.build();
+ }
+
+ private static String recordToJsonStatic(Record record, String shardId, boolean shouldAddOffsets) {
+ String dataStr = record.data().asUtf8String();
+ // Pure empty or null records in Kinesis is not meaningful.
+ if (dataStr == null || dataStr.trim().isEmpty()) {
+ return null;
+ }
+ if (shouldAddOffsets) {
+ try {
+ ObjectNode node = (ObjectNode) OBJECT_MAPPER.readTree(dataStr);
+ node.put("_hoodie_kinesis_source_sequence_number", record.sequenceNumber());
+ node.put("_hoodie_kinesis_source_shard_id", shardId);
+ node.put("_hoodie_kinesis_source_partition_key", record.partitionKey());
+ if (record.approximateArrivalTimestamp() != null) {
+ node.put("_hoodie_kinesis_source_timestamp",
+ record.approximateArrivalTimestamp().toEpochMilli());
+ }
+ return OBJECT_MAPPER.writeValueAsString(node);
+ } catch (Exception e) {
+ return dataStr;
+ }
+ }
+ return dataStr;
+ }
+
+ private Map<String, String> buildCheckpointFromSummaries(List<ShardFetchSummary> summaries) {
+ Map<String, String> checkpoint = new HashMap<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.getLastSequenceNumber().isPresent()) {
+ checkpoint.put(s.getShardId(), s.getLastSequenceNumber().get());
+ }
+ }
+ return checkpoint;
+ }
+
+ @Override
+ protected String createCheckpointFromBatch(JavaRDD<String> batch,
+ KinesisOffsetGen.KinesisShardRange[] shardRangesRead,
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges) {
+ // Build checkpoint from ALL shards so filtered-out (fully consumed) shards are preserved.
+ // Otherwise the next run would omit them from the checkpoint and re-read from TRIM_HORIZON.
+ Map<String, String> fullCheckpoint = new HashMap<>();
+ for (KinesisOffsetGen.KinesisShardRange range : allShardRanges) {
+ String lastSeq;
+ String endSeq;
+ if (lastCheckpointData != null && lastCheckpointData.containsKey(range.getShardId())) {
+ // Shard we read from: use data from the read
+ lastSeq = lastCheckpointData.get(range.getShardId());
+ endSeq = range.getEndingSequenceNumber().orElse(null);
+ } else {
+ // Filtered shard (not read): preserve from range so next run won't re-read
+ lastSeq = range.getStartingSequenceNumber().orElse("");
+ endSeq = range.getEndingSequenceNumber().orElse(null);
+ }
+ // for test only
+ // LocalStack returns Long.MAX_VALUE for closed shards; use lastSeq as endSeq so we can detect
+ // "fully consumed" when the parent shard expires (lastSeq >= endSeq).
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && lastSeq != null && !lastSeq.isEmpty()) {
+ endSeq = lastSeq;
+ }
Review Comment:
will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2898226222
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
Review Comment:
will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4015431158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a28e9e73447bf984536783a76949fd1f98f531dc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227)
* 759d3cdbf5684af44f12fb2ec7ef40c38760ab23 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034974484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550)
* 9af3386e49cb4c09581bce9632b306b2b4ced5de UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3938627562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930623240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129)
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2909724150
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034821050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913158395
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
Review Comment:
These are RuntimeExceptions; normally we don't catch runtime errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4028294280
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513)
* 8d77ec07f756f71fb3105c8a38008a995b05d793 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034496716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 8d77ec07f756f71fb3105c8a38008a995b05d793 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517)
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937682182
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a517e2685c28a1101af008e73abfe04522484a17 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166)
* 82ade47dcdb28b4fd8db26ff39980c5e72fca27b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324094743
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.63%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 19 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.63% -21.26%
+ Complexity 28537 16589 -11948
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 54424 -39898
- Misses 34996 54093 +19097
+ Partials 7592 5729 -1863
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.76% <0.00%> (-16.72%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324088606
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.85%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 23 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 45.85% -23.04%
+ Complexity 28537 15290 -13247
=============================================
Files 2480 1979 -501
Lines 136910 109262 -27648
Branches 16679 13546 -3133
=============================================
- Hits 94322 50107 -44215
- Misses 34996 53948 +18952
+ Partials 7592 5207 -2385
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.57% <0.00%> (-12.71%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1622 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324192104
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.32%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.32% -2.57%
+ Complexity 28537 23155 -5382
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75779 -18543
+ Misses 34996 31695 -3301
+ Partials 7592 6772 -820
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.14% <0.00%> (-0.33%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.99% <0.00%> (-0.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [832 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324197872
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.33%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.33% -2.57%
+ Complexity 28537 23155 -5382
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75782 -18540
+ Misses 34996 31693 -3303
+ Partials 7592 6771 -821
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.99% <0.00%> (-0.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [831 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324110808
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.68%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.68% -21.22%
+ Complexity 28537 16602 -11935
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 54476 -39846
- Misses 34996 54047 +19051
+ Partials 7592 5723 -1869
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.40% <0.00%> (-17.08%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324865560
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.61%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.61% -2.28%
+ Complexity 28537 23251 -5286
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76106 -18216
+ Misses 34996 31359 -3637
+ Partials 7592 6781 -811
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.30%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [820 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325578825
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.74%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 21 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.74% -21.15%
+ Complexity 28537 16618 -11919
=============================================
Files 2480 2026 -454
Lines 136910 114421 -22489
Branches 16679 14664 -2015
=============================================
- Hits 94322 54631 -39691
- Misses 34996 53935 +18939
+ Partials 7592 5855 -1737
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.80% <0.00%> (-16.67%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.73% <0.00%> (-12.55%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1579 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326344408
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 25.09%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`44f0050`](https://app.codecov.io/gh/apache/hudi/commit/44f0050b30e808c35359269a6da7aa6bad6f926e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (44f0050). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (44f0050) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 25.09% -43.80%
+ Complexity 28537 10357 -18180
=============================================
Files 2480 2432 -48
Lines 136910 130602 -6308
Branches 16679 15205 -1474
=============================================
- Hits 94322 32774 -61548
- Misses 34996 94699 +59703
+ Partials 7592 3129 -4463
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `9.32% <0.00%> (-35.10%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1741 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326253228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* db19834cf6915d0a9e1674ed36c04c2a45d61545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640)
* 44f0050b30e808c35359269a6da7aa6bad6f926e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326473071
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.75%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b477867`](https://app.codecov.io/gh/apache/hudi/commit/b477867a5091d193bc749516cc4fc0bc32d2a591?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (b477867). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (b477867) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.75% -24.14%
+ Complexity 28537 8501 -20036
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28031 -66291
+ Misses 34996 31519 -3477
+ Partials 7592 3076 -4516
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.75% <ø> (-0.07%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327047737
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.92%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`937bbb2`](https://app.codecov.io/gh/apache/hudi/commit/937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (937bbb2). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (937bbb2) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 53.92% -14.97%
+ Complexity 28537 12391 -16146
=============================================
Files 2480 1433 -1047
Lines 136910 71927 -64983
Branches 16679 8203 -8476
=============================================
- Hits 94322 38788 -55534
+ Misses 34996 29659 -5337
+ Partials 7592 3480 -4112
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.81% <ø> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.13%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1830 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327908018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
}, {
"hash" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* 7af0201256fc80be36c78bc2e56df221b876902a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650)
* a193e7e13b56a25526e18c3eec95cbdf3d16c79c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324180991
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.12%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.12% -2.77%
+ Complexity 28537 23037 -5500
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75544 -18778
+ Misses 34996 31982 -3014
+ Partials 7592 6720 -872
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.92% <0.00%> (-0.56%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <0.00%> (-0.45%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [847 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325700840
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 63.66%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 9 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|8|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 63.66% -5.24%
+ Complexity 28537 22427 -6110
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 74423 -19899
- Misses 34996 35425 +429
+ Partials 7592 7057 -535
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.50% <0.00%> (-3.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.91% <0.00%> (-0.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1038 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327732086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* 7af0201256fc80be36c78bc2e56df221b876902a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185264812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 5aaba50d2b78079d344e8d39c6baf0e76da9f5ae Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557)
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034092874
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
Review Comment:
Could you add a javadoc to explain this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034799278
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisReadConfig.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+
+import java.io.Serializable;
+
+/**
+ * Serializable configuration for Kinesis reads, used in Spark closures to avoid
+ * capturing non-serializable KinesisOffsetGen.
+ */
+@AllArgsConstructor
+@Getter
+public class KinesisReadConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final String streamName;
+ private final String region;
+ private final String endpointUrl; // null if not set
+ private final String accessKey; // null if not set
+ private final String secretKey; // null if not set
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final boolean shouldAddOffsets;
+ private final boolean enableDeaggregation;
+ private final int maxRecordsPerRequest;
+ private final long intervalMs;
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3036400342
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(PREFIX + "max.events")
+ .defaultValue(5000000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "partitions")
+ .defaultValue(0L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.fail.on.data.loss")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
+ .defaultValue(10000)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
+
+ public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "get_records.interval.ms")
+ .defaultValue(200L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
+
+ public static final ConfigProperty<Long> KINESIS_RETRY_INITIAL_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "retry.initial_interval_ms")
+ .defaultValue(1000L)
+ .sinceVersion("1.2.0")
Review Comment:
🤖 nit: config key uses underscores in `initial_interval_ms`; use dots instead (e.g., `retry.initial.interval.ms`) to match the Hudi convention throughout this file.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(PREFIX + "max.events")
+ .defaultValue(5000000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "partitions")
+ .defaultValue(0L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.fail.on.data.loss")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
+ .defaultValue(10000)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
+
Review Comment:
🤖 nit: config key uses underscore in `get_records.interval.ms`; the convention throughout KinesisSourceConfig and Hudi is dots (e.g., `stream.name`, `enable.deaggregation`). consider `get.records.interval.ms` for consistency.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(PREFIX + "max.events")
+ .defaultValue(5000000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "partitions")
+ .defaultValue(0L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.fail.on.data.loss")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
+ .defaultValue(10000)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
+
+ public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "get_records.interval.ms")
+ .defaultValue(200L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
+
+ public static final ConfigProperty<Long> KINESIS_RETRY_INITIAL_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "retry.initial_interval_ms")
+ .defaultValue(1000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Initial backoff in ms when Kinesis returns ProvisionedThroughputExceededException. "
+ + "Backoff doubles each retry up to retry.max_interval_ms.");
+
+ public static final ConfigProperty<Long> KINESIS_RETRY_MAX_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "retry.max_interval_ms")
+ .defaultValue(10000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum backoff in ms between retries for throughput exceeded.");
+
+ public static final ConfigProperty<Long> KINESIS_THROTTLE_TIMEOUT_MS = ConfigProperty
+ .key(PREFIX + "retry.throttle_timeout_ms")
+ .defaultValue(600000L)
+ .sinceVersion("1.2.0")
Review Comment:
🤖 nit: config key uses underscores in `throttle_timeout_ms`; use dots instead (e.g., `retry.throttle.timeout.ms`) to match the Hudi convention throughout this file.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/KinesisTestUtils.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.testutils;
+
+import com.amazonaws.kinesis.agg.AggRecord;
+import com.amazonaws.kinesis.agg.RecordAggregator;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.ScalingType;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+import software.amazon.awssdk.services.kinesis.model.UpdateShardCountRequest;
+
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.nio.ByteBuffer;
+import java.net.URI;
+import java.util.List;
+
+/**
+ * Test utilities for Kinesis Data Streams using LocalStack.
+ */
+public class KinesisTestUtils {
+ private static final DockerImageName LOCALSTACK_IMAGE =
+ DockerImageName.parse("localstack/localstack:4.1.0");
+
+ private LocalStackContainer localStack;
+ private KinesisClient kinesisClient;
+
+ public KinesisTestUtils setup() {
+ localStack = new LocalStackContainer(LOCALSTACK_IMAGE)
+ .withServices(LocalStackContainer.Service.KINESIS);
+ localStack.start();
+ kinesisClient = KinesisClient.builder()
+ .endpointOverride(localStack.getEndpointOverride(LocalStackContainer.Service.KINESIS))
+ .credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(localStack.getAccessKey(), localStack.getSecretKey())))
+ .region(Region.of(localStack.getRegion()))
+ .build();
+ return this;
+ }
+
+ public String getEndpointUrl() {
+ if (localStack == null || !localStack.isRunning()) {
+ throw new IllegalStateException("LocalStack container is not running. Please start the container first.");
+ }
+ URI endpoint = localStack.getEndpointOverride(LocalStackContainer.Service.KINESIS);
+ return endpoint.toString();
+ }
+
+ public String getRegion() {
+ if (localStack == null || !localStack.isRunning()) {
+ throw new IllegalStateException("LocalStack container is not running. Please start the container first.");
+ }
+ return localStack.getRegion();
+ }
+
+ public String getAccessKey() {
+ if (localStack == null || !localStack.isRunning()) {
+ throw new IllegalStateException("LocalStack container is not running. Please start the container first.");
+ }
+ return localStack.getAccessKey();
+ }
+
+ public String getSecretKey() {
+ if (localStack == null || !localStack.isRunning()) {
+ throw new IllegalStateException("LocalStack container is not running. Please start the container first.");
+ }
+ return localStack.getSecretKey();
+ }
+
+ public void createStream(String streamName) {
+ createStream(streamName, 1);
+ }
+
+ public void createStream(String streamName, int shardCount) {
+ kinesisClient.createStream(
+ CreateStreamRequest.builder()
+ .streamName(streamName)
+ .shardCount(shardCount)
+ .build());
+ waitForStreamActive(streamName);
+ }
+
+ private void waitForStreamActive(String streamName) {
+ waitForStreamActive(streamName, 30);
+ }
+
+ private void waitForStreamActive(String streamName, int maxAttempts) {
+ try {
+ for (int i = 0; i < maxAttempts; i++) {
+ String status = kinesisClient.describeStream(
+ DescribeStreamRequest.builder().streamName(streamName).build())
+ .streamDescription().streamStatus().toString();
+ if (StreamStatus.ACTIVE.toString().equals(status)) {
+ return;
+ }
+ Thread.sleep(500);
+ }
+ throw new RuntimeException("Stream " + streamName + " did not become ACTIVE in time");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for stream", e);
+ }
+ }
+
+ public void sendRecords(String streamName, List<String> records) {
+ for (String record : records) {
+ kinesisClient.putRecord(
+ PutRecordRequest.builder()
+ .streamName(streamName)
+ .partitionKey(String.valueOf(System.nanoTime()))
+ .data(SdkBytes.fromUtf8String(record))
+ .build());
+ }
+ }
+
+ public void sendRecords(String streamName, String[] records) {
+ for (String record : records) {
+ kinesisClient.putRecord(
+ PutRecordRequest.builder()
+ .streamName(streamName)
+ .partitionKey(String.valueOf(System.nanoTime()))
Review Comment:
🤖 nit: the `sendRecords(List)` and `sendRecords(String[])` methods have identical loop implementations—have you considered consolidating to just the List version?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/KinesisTestUtils.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.testutils;
+
+import com.amazonaws.kinesis.agg.AggRecord;
+import com.amazonaws.kinesis.agg.RecordAggregator;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.DescribeStreamRequest;
+import software.amazon.awssdk.services.kinesis.model.PutRecordRequest;
+import software.amazon.awssdk.services.kinesis.model.ScalingType;
+import software.amazon.awssdk.services.kinesis.model.StreamStatus;
+import software.amazon.awssdk.services.kinesis.model.UpdateShardCountRequest;
+
+import org.testcontainers.containers.localstack.LocalStackContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.nio.ByteBuffer;
+import java.net.URI;
+import java.util.List;
+
+/**
+ * Test utilities for Kinesis Data Streams using LocalStack.
+ */
+public class KinesisTestUtils {
+ private static final DockerImageName LOCALSTACK_IMAGE =
+ DockerImageName.parse("localstack/localstack:4.1.0");
+
+ private LocalStackContainer localStack;
+ private KinesisClient kinesisClient;
+
+ public KinesisTestUtils setup() {
+ localStack = new LocalStackContainer(LOCALSTACK_IMAGE)
+ .withServices(LocalStackContainer.Service.KINESIS);
+ localStack.start();
+ kinesisClient = KinesisClient.builder()
+ .endpointOverride(localStack.getEndpointOverride(LocalStackContainer.Service.KINESIS))
+ .credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(localStack.getAccessKey(), localStack.getSecretKey())))
+ .region(Region.of(localStack.getRegion()))
+ .build();
+ return this;
+ }
+
+ public String getEndpointUrl() {
+ if (localStack == null || !localStack.isRunning()) {
+ throw new IllegalStateException("LocalStack container is not running. Please start the container first.");
Review Comment:
🤖 nit: could you extract the `if (localStack == null || !localStack.isRunning())` check that repeats in `getEndpointUrl()`, `getRegion()`, `getAccessKey()`, and `getSecretKey()` into a private helper?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(PREFIX + "max.events")
+ .defaultValue(5000000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "partitions")
+ .defaultValue(0L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.fail.on.data.loss")
+ .defaultValue(false)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
+ .defaultValue(10000)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
+
+ public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "get_records.interval.ms")
+ .defaultValue(200L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
+
+ public static final ConfigProperty<Long> KINESIS_RETRY_INITIAL_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "retry.initial_interval_ms")
+ .defaultValue(1000L)
+ .sinceVersion("1.2.0")
+ .markAdvanced()
+ .withDocumentation("Initial backoff in ms when Kinesis returns ProvisionedThroughputExceededException. "
+ + "Backoff doubles each retry up to retry.max_interval_ms.");
+
+ public static final ConfigProperty<Long> KINESIS_RETRY_MAX_INTERVAL_MS = ConfigProperty
+ .key(PREFIX + "retry.max_interval_ms")
+ .defaultValue(10000L)
+ .sinceVersion("1.2.0")
Review Comment:
🤖 nit: config key uses underscores in `max_interval_ms`; use dots instead (e.g., `retry.max.interval.ms`) to match the Hudi convention throughout this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034848777
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
Review Comment:
this variable has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034355541
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisReadConfig.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+
+import java.io.Serializable;
+
+/**
+ * Serializable configuration for Kinesis reads, used in Spark closures to avoid
+ * capturing non-serializable KinesisOffsetGen.
+ */
+@AllArgsConstructor
+@Getter
+public class KinesisReadConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final String streamName;
+ private final String region;
+ private final String endpointUrl; // null if not set
+ private final String accessKey; // null if not set
+ private final String secretKey; // null if not set
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final boolean shouldAddOffsets;
+ private final boolean enableDeaggregation;
+ private final int maxRecordsPerRequest;
+ private final long intervalMs;
Review Comment:
Let's prefix `intervalMs`, explaining what this interval is for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034782773
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034269052
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
Review Comment:
For now, the RDD is repartitioned with the parallelism which is good enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034044229
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
Review Comment:
Looks like this is missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034145953
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
Review Comment:
Let's address this in an immediate follow-up. My concern is that not all auth modes are supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930431484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930419271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930667050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129)
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* fec6001976389d10a38feda0a413fb6043f23ace Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930857071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c44c273366ff265822001c38f3a04f427085e58 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129)
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* fec6001976389d10a38feda0a413fb6043f23ace Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134)
* d09affde11485fedf2aba0f93230da54669bf307 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913620620
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
Review Comment:
when a shard was created, merged/splitted before the next run; then this shard becomes closed before any read. For it, the `lastSeq` is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4035039332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550)
* 9af3386e49cb4c09581bce9632b306b2b4ced5de Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554)
* 5aaba50d2b78079d344e8d39c6baf0e76da9f5ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034985973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550)
* 9af3386e49cb4c09581bce9632b306b2b4ced5de Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2872771179
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ if (minPartitions > 0 && minPartitions > shardRanges.length) {
+ int targetPartitions = (int) minPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (minPartitions={})",
+ shardRanges.length, targetPartitions, minPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+
+ // Information for checkpoint generation.
+ List<ShardFetchSummary> summaries = fetchRdd
+ .map(r -> new ShardFetchSummary(r.getShardId(), r.getLastSequenceNumber(), r.getRecords().size(),
+ r.isReachedEndOfShard()))
+ .collect();
+ lastCheckpointData = buildCheckpointFromSummaries(summaries);
+ Set<String> reached = new HashSet<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.isReachedEndOfShard()) {
+ reached.add(s.getShardId());
+ }
+ }
+ shardsReachedEnd = reached;
+ lastRecordCount = summaries.stream().mapToLong(ShardFetchSummary::getRecordCount).sum();
+
+ return recordRdd;
+ }
+
+ private static KinesisClient createKinesisClientFromConfig(KinesisReadConfig config) {
+ software.amazon.awssdk.services.kinesis.KinesisClientBuilder builder =
+ KinesisClient.builder().region(software.amazon.awssdk.regions.Region.of(config.getRegion()));
+ if (config.getEndpointUrl() != null && !config.getEndpointUrl().isEmpty()) {
+ builder = builder.endpointOverride(java.net.URI.create(config.getEndpointUrl()));
+ }
+ if (config.getAccessKey() != null && !config.getAccessKey().isEmpty()
+ && config.getSecretKey() != null && !config.getSecretKey().isEmpty()) {
+ builder = builder.credentialsProvider(
+ software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
+ software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create(
+ config.getAccessKey(), config.getSecretKey())));
+ }
+ return builder.build();
+ }
+
+ private static String recordToJsonStatic(Record record, String shardId, boolean shouldAddOffsets) {
+ String dataStr = record.data().asUtf8String();
+ // Pure empty or null records in Kinesis is not meaningful.
+ if (dataStr == null || dataStr.trim().isEmpty()) {
+ return null;
+ }
+ if (shouldAddOffsets) {
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2874634649
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
+ .key(PREFIX + "getRecords.maxRecords")
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324057402
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 282c1bea218876530b319f14dedd7a75a3fff469 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324086446
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.84%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|2|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 45.84% -23.05%
+ Complexity 28537 15290 -13247
=============================================
Files 2480 1979 -501
Lines 136910 109257 -27653
Branches 16679 13539 -3140
=============================================
- Hits 94322 50090 -44232
- Misses 34996 53957 +18961
+ Partials 7592 5210 -2382
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <0.00%> (-12.73%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1628 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324456250
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 282c1bea218876530b319f14dedd7a75a3fff469 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624)
* 08ae17785490db6a2b9432acbc3196517ed73845 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324941855
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.62%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 9 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.62% -2.27%
+ Complexity 28537 23253 -5284
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76115 -18207
+ Misses 34996 31353 -3643
+ Partials 7592 6778 -814
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.22% <0.00%> (-0.25%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [817 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324119912
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.71%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.71% -21.19%
+ Complexity 28537 16613 -11924
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 54510 -39812
- Misses 34996 54018 +19022
+ Partials 7592 5718 -1874
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.99% <0.00%> (-17.48%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324704345
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.67%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.67% -21.22%
+ Complexity 28537 16609 -11928
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 54472 -39850
- Misses 34996 54050 +19054
+ Partials 7592 5724 -1868
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.39% <0.00%> (-17.08%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325108192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 08ae17785490db6a2b9432acbc3196517ed73845 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633)
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325606205
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.08%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|4|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.08% -20.82%
+ Complexity 28537 16778 -11759
=============================================
Files 2480 2026 -454
Lines 136910 114421 -22489
Branches 16679 14664 -2015
=============================================
- Hits 94322 55016 -39306
- Misses 34996 53478 +18482
+ Partials 7592 5927 -1665
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.44% <0.00%> (-17.03%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.65% <0.00%> (-12.63%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1576 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325782363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* db19834cf6915d0a9e1674ed36c04c2a45d61545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325811372
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.81%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.81% -3.09%
+ Complexity 28537 23512 -5025
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76939 -17383
+ Misses 34996 32961 -2035
+ Partials 7592 7005 -587
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.61% <0.00%> (-0.87%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [860 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326317949
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 9.32%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`44f0050`](https://app.codecov.io/gh/apache/hudi/commit/44f0050b30e808c35359269a6da7aa6bad6f926e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (44f0050). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (44f0050) |
>|------|------|------|
>|hadoop-mr-java-client|1|0|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 9.32% -59.57%
+ Complexity 28537 3982 -24555
=============================================
Files 2480 2432 -48
Lines 136910 130602 -6308
Branches 16679 15205 -1474
=============================================
- Hits 94322 12182 -82140
- Misses 34996 117422 +82426
+ Partials 7592 998 -6594
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `9.32% <0.00%> (-35.10%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1992 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327413161
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.02%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|4|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.02% -20.87%
+ Complexity 28537 16765 -11772
=============================================
Files 2480 2026 -454
Lines 136910 114421 -22489
Branches 16679 14664 -2015
=============================================
- Hits 94322 54954 -39368
- Misses 34996 53526 +18530
+ Partials 7592 5941 -1651
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.69% <0.00%> (-16.79%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.64% <0.00%> (-12.64%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1579 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327372395
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.15%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|2|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 46.15% -22.75%
+ Complexity 28537 15445 -13092
=============================================
Files 2480 1991 -489
Lines 136910 109597 -27313
Branches 16679 13610 -3069
=============================================
- Hits 94322 50581 -43741
- Misses 34996 53659 +18663
+ Partials 7592 5357 -2235
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.95% <0.00%> (-12.33%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1647 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327443067
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.00%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 55.00% -13.90%
+ Complexity 28537 19229 -9308
=============================================
Files 2480 2026 -454
Lines 136910 114421 -22489
Branches 16679 14664 -2015
=============================================
- Hits 94322 62935 -31387
- Misses 34996 44954 +9958
+ Partials 7592 6532 -1060
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.77% <0.00%> (-16.71%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.68% <0.00%> (-0.60%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1410 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329034942
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.60%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.60% -21.29%
+ Complexity 28537 16992 -11545
=============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15204 -1475
=============================================
- Hits 94322 55654 -38668
- Misses 34996 55122 +20126
+ Partials 7592 6129 -1463
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.78% <0.00%> (-16.69%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <0.00%> (-12.73%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1572 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329028178
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.59%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.59% -21.30%
+ Complexity 28537 16990 -11547
=============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15204 -1475
=============================================
- Hits 94322 55644 -38678
- Misses 34996 55123 +20127
+ Partials 7592 6138 -1454
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.74% <0.00%> (-16.73%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <0.00%> (-12.73%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1573 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3946633013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 917c414dba55734c588d3c43c4b0e16e137fb211 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216)
* a28e9e73447bf984536783a76949fd1f98f531dc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3938557168
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 45.42%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d43f8eb`](https://app.codecov.io/gh/apache/hudi/commit/d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 984 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (d43f8eb). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (d43f8eb) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 61.43% 45.42% -16.01%
+ Complexity 23082 8372 -14710
=============================================
Files 2108 1177 -931
Lines 127636 60233 -67403
Branches 14534 6414 -8120
=============================================
- Hits 78409 27363 -51046
+ Misses 42873 29897 -12976
+ Partials 6354 2973 -3381
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.42% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 3243 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3944854520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182)
* 917c414dba55734c588d3c43c4b0e16e137fb211 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2835540302
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -3112,6 +3119,281 @@ public void testJsonKafkaDFSSource() throws Exception {
deltaStreamer.shutdownGracefully();
}
+ /**
+ * Tests DeltaStreamer ingestion from JsonKinesisSource using LocalStack.
+ */
+ @Nested
+ @org.junit.jupiter.api.TestInstance(org.junit.jupiter.api.TestInstance.Lifecycle.PER_CLASS)
+ class TestKinesisSource {
+
+ private KinesisTestUtils kinesisTestUtils;
+
+ @BeforeAll
+ void initKinesis() {
+ kinesisTestUtils = new KinesisTestUtils().setup();
+ }
+
+ @AfterAll
+ void tearDownKinesis() {
+ if (kinesisTestUtils != null) {
+ kinesisTestUtils.teardown();
+ }
+ }
+
+ @Test
+ public void testJsonKinesisDFSSource() throws Exception {
+ String streamName = "test-kinesis-stream-" + testNum;
+ // Create stream with at least 2 shards to exercise multi-shard reading
+ prepareJsonKinesisDFSFiles(JSON_KINESIS_NUM_RECORDS, true, streamName);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, JsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+
+ // Track commit completion times for incremental queries
+ List<String> commitCompletionTimes = new ArrayList<>();
+
+ // Batch 1: initial ingestion
+ deltaStreamer.sync();
+ assertRecordCount(JSON_KINESIS_NUM_RECORDS, tableBasePath, sqlContext);
+ String commit1 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit1, JSON_KINESIS_NUM_RECORDS);
+
+ int totalRecords = JSON_KINESIS_NUM_RECORDS;
+ // Batch 2
+ int batch2Records = 10;
+ totalRecords += batch2Records;
+ prepareJsonKinesisDFSFiles(batch2Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit2 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit2, batch2Records);
+
+ // Batch 3
+ int batch3Records = 8;
+ totalRecords += batch3Records;
+ prepareJsonKinesisDFSFiles(batch3Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit3 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit3, batch3Records);
+
+ // Batch 4
+ int batch4Records = 7;
+ totalRecords += batch4Records;
+ prepareJsonKinesisDFSFiles(batch4Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit4 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit4, batch4Records);
+ deltaStreamer.shutdownGracefully();
+
+ // Incremental queries for a subset of commits
+ sqlContext.clearCache();
+ String c1 = commitCompletionTimes.get(0);
+ String c2 = commitCompletionTimes.get(1);
+ String c3 = commitCompletionTimes.get(2);
+ String c4 = commitCompletionTimes.get(3);
+
+ // Incremental: first commit only (batch 1)
+ Dataset<Row> incrBatch1 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), "000")
+ .option(DataSourceReadOptions.END_COMMIT().key(), c1)
+ .load(tableBasePath);
+ assertEquals(JSON_KINESIS_NUM_RECORDS, incrBatch1.count(), "Incremental read for batch 1");
+
+ // Incremental: commits 2 and 3 (batches 2 + 3)
+ Dataset<Row> incrBatches2And3 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), c1)
+ .option(DataSourceReadOptions.END_COMMIT().key(), c3)
+ .load(tableBasePath);
+ assertEquals(batch2Records + batch3Records, incrBatches2And3.count(), "Incremental read for batches 2 and 3");
+
+ // Incremental: commits 3 and 4 (batches 3 + 4)
+ Dataset<Row> incrBatches3And4 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), c2)
+ .option(DataSourceReadOptions.END_COMMIT().key(), c4)
+ .load(tableBasePath);
+ assertEquals(batch3Records + batch4Records, incrBatches3And4.count(), "Incremental read for batches 3 and 4");
+
+ // Verify data correctness: read back and check _row_key exists
+ sqlContext.clearCache();
+ Dataset<Row> ds = sqlContext.read().format("org.apache.hudi").load(tableBasePath);
+ assertEquals(totalRecords, ds.count());
+ assertTrue(ds.filter("_row_key is not null").count() > 0);
+ }
+
+ @Test
+ public void testJsonKinesisAggregatedRecords() throws Exception {
+ String streamName = "test-kinesis-stream-agg-" + testNum;
+ kinesisTestUtils.createStream(streamName, 2);
+ HoodieTestDataGenerator dataGenerator = new HoodieTestDataGenerator(System.nanoTime());
+ String[] jsonRecords = UtilitiesTestBase.Helpers.jsonifyRecords(
+ dataGenerator.generateInsertsAsPerSchema("000", 6, HoodieTestDataGenerator.TRIP_SCHEMA));
+ kinesisTestUtils.sendAggregatedRecords(streamName, jsonRecords);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_agg_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, JsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+ deltaStreamer.sync();
+ assertRecordCount(6, tableBasePath, sqlContext);
+ deltaStreamer.shutdownGracefully();
+ testNum++;
+ }
+
+ @Test
+ public void testJsonKinesisShardSplitCheckpoint() throws Exception {
+ String streamName = "test-kinesis-stream-split-" + testNum;
+ kinesisTestUtils.createStream(streamName, 2);
+ prepareJsonKinesisDFSFiles(10, false, streamName);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_split_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, JsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+ deltaStreamer.sync();
+ assertRecordCount(10, tableBasePath, sqlContext);
+ String checkpointAfterBatch1 = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterBatch1);
+ assertTrue(checkpointAfterBatch1.startsWith(streamName + ","));
+ assertTrue(checkpointAfterBatch1.contains(":"), "Checkpoint should have shard:seq format");
+
+ kinesisTestUtils.updateShardCount(streamName, 4);
+ prepareJsonKinesisDFSFiles(5, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(15, tableBasePath, sqlContext);
+ String checkpointAfterSplit = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterSplit);
+ assertTrue(checkpointAfterSplit.startsWith(streamName + ","));
+ // Closed parent shards must have lastSeq|endSeq with endSeq <= lastSeq so we can detect
+ // "fully consumed" when parent expires. LocalStack returns Long.MAX_VALUE; we replace with lastSeq.
+ assertFalse(checkpointAfterSplit.contains("9223372036854775807"),
+ "Checkpoint should not contain Long.MAX_VALUE as endSeq (parent expiry would fail)");
+ int initialShardCount = KinesisOffsetGen.CheckpointUtils.strToOffsets(checkpointAfterBatch1).size();
+ int shardCountAfterSplit = KinesisOffsetGen.CheckpointUtils.strToOffsets(checkpointAfterSplit).size();
+ assertTrue(shardCountAfterSplit > initialShardCount,
+ "Checkpoint after split should have more shards (" + shardCountAfterSplit
+ + ") than initial (" + initialShardCount + ")");
+ deltaStreamer.shutdownGracefully();
+ testNum++;
+ }
+
+ @Test
+ public void testJsonKinesisShardMergeCheckpoint() throws Exception {
+ String streamName = "test-kinesis-stream-merge-" + testNum;
+ kinesisTestUtils.createStream(streamName, 4);
+ prepareJsonKinesisDFSFiles(8, false, streamName);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_merge_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, JsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+ deltaStreamer.sync();
+ assertRecordCount(8, tableBasePath, sqlContext);
+ String checkpointAfterBatch1 = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterBatch1);
+
+ kinesisTestUtils.updateShardCount(streamName, 2);
+ prepareJsonKinesisDFSFiles(4, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(12, tableBasePath, sqlContext);
+ String checkpointAfterMerge = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterMerge);
+ assertTrue(checkpointAfterMerge.startsWith(streamName + ","));
+ int initialShardCount = KinesisOffsetGen.CheckpointUtils.strToOffsets(checkpointAfterBatch1).size();
+ int shardCountAfterMerge = KinesisOffsetGen.CheckpointUtils.strToOffsets(checkpointAfterMerge).size();
+ assertTrue(shardCountAfterMerge > initialShardCount,
Review Comment:
After merge, new child shards are generated, but the parent shards are still there and not expired yet. So ">" not "<"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2849377967
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
Review Comment:
Could `hasUnreadRecords` be reused here?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
Review Comment:
`lastSeq` should not be empty, otherwise it should throw an error. It would be good to add `ValidationUtils#checkArgument`.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
Review Comment:
Why could `lastSeq` be larger than `endSeq`? I assume `lastSeq` should always be less than or equal to `endSeq`.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
Review Comment:
The `minPartitions` or `targetParallelism` is not used to determine the parallelism or the `ranges` like Kafka source. Do we want to consider that in case the number of shards is low?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2853980457
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
+ }
+
+ T batch = toBatch(shardRanges);
+ // Pass allShardRanges so filtered-out shards are preserved in the checkpoint; otherwise
+ // next run would re-read them from TRIM_HORIZON and cause duplicates
+ String checkpointStr = createCheckpointFromBatch(batch, shardRanges, allShardRanges);
+ long totalMsgs = getRecordCount(batch);
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, totalMsgs);
+
+ log.info("Read {} records from Kinesis stream {} with {} shards, checkpoint: {}",
+ totalMsgs, offsetGen.getStreamName(), shardRanges.length, checkpointStr);
+
+ return new InputBatch<>(Option.of(batch), checkpointStr);
+ }
+
+ protected abstract T toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges);
+
+ /**
+ * Create checkpoint string from the batch and shard ranges.
+ * Subclasses provide checkpoint data (shardId -> sequenceNumber) collected during the read.
+ * Must include both read shards (from shardRangesRead) and filtered shards (from allShardRanges)
+ * so the next run does not re-read filtered-out shards from TRIM_HORIZON.
+ */
+ protected abstract String createCheckpointFromBatch(T batch,
+ KinesisOffsetGen.KinesisShardRange[] shardRangesRead,
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges);
Review Comment:
Could `shardRangesRead` and `allShardRanges` be renamed to be easily understood?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
Review Comment:
To clarify, `offsetGen.getNextShardRanges` returns all the open and closed ranges (excluding expired ranges) from the checkpoint and current correct?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
Review Comment:
There should also be validation on open and closed shards. If the last sequence number of a shard in last checkpoint is before the start of the shard based on the current state, e.g., due to data retention, there is also data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2912848101
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2912878617
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
Review Comment:
Yes, the region parameter is necessary to access a Kinesis Data Stream because Kinesis is a regional service, and streams are scoped to a specific AWS Region and account.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2912858173
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4028101827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2911635834
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisDeaggregator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * De-aggregates KPL (Kinesis Producer Library) aggregated records into individual user records.
+ * Non-aggregated records are returned unchanged.
+ */
+public final class KinesisDeaggregator {
+
+ private KinesisDeaggregator() {
+ }
+
+ /**
+ * De-aggregate SDK v2 Kinesis records. Aggregated records (from KPL) are split into user records.
+ * Non-aggregated records pass through unchanged.
+ */
+ public static List<Record> deaggregate(List<Record> records) {
+ if (records == null || records.isEmpty()) {
+ return new ArrayList<>();
+ }
+ List<com.amazonaws.services.kinesis.model.Record> v1Records = new ArrayList<>(records.size());
+ for (Record r : records) {
+ v1Records.add(toV1Record(r));
+ }
+ List<UserRecord> userRecords = UserRecord.deaggregate(v1Records);
+ List<Record> result = new ArrayList<>(userRecords.size());
+ for (UserRecord ur : userRecords) {
+ result.add(toV2Record(ur));
+ }
Review Comment:
`UserRecord.deaggregate` is the only known way to do the de-aggregation, which requires V2 (regular records) to V1 and back to V2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913220023
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
Review Comment:
> Is this for paginated responses? Is there an API handling this for reuse, instead of hand-crafting the logic again?
1. yes, 2. No, after I searched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2911401602
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
Review Comment:
Currently, When there is nothing to ingestion,
CASE 1: this is new stream, the checkpointStr will be empty
CASE 2: this is not a new stream, the last checkpoint str can be used.
In either case, we keep using the current checkpoint: either empty or last one.
When is A range defined as having no unread records:
CASE 1: range is closed, and lastSeqNum >= endSeqNum. In this case, this shard has been fully consumed; normally we don't need to update the last checkpoint.
CASE 2: range is closed, and lastSeqNum = null, which means the shard has not been consumed before. Only If LATEST strategy is used, this shard will never be consumed; no need to add it to the checkpoint.
Therefore, we should be fine for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2914638180
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.Arrays;
+import java.util.Map;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddOffsets;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddOffsets = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ KinesisOffsetGen.KinesisShardRange[] shardRanges = offsetGen.getNextShardRanges(
+ lastCheckpoint, sourceLimit, metrics);
+
+ // Filter out shards with no unread records to avoid unnecessary GetRecords calls
+ boolean useLatestWhenNoCheckpoint =
+ offsetGen.getStartingPosition() == KinesisSourceConfig.KinesisStartingPosition.LATEST;
+ KinesisOffsetGen.KinesisShardRange[] allShardRanges = shardRanges;
+ int beforeFilter = shardRanges.length;
+ shardRanges = Arrays.stream(shardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestWhenNoCheckpoint))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (beforeFilter > shardRanges.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ beforeFilter - shardRanges.length, shardRanges.length);
+ }
+
+ if (shardRanges.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
Review Comment:
> Could there be a case where the checkpoint needs to be set, instead of empty string, and there is no message to ingest?
AFAIK, we don't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4034640339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 8d77ec07f756f71fb3105c8a38008a995b05d793 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517)
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 7e41c5fd112cbd47e9554bbf27e6d23c2f6de149 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3033762221
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
Review Comment:
Right now, without caching the RDD (turning off the flag), the data is read twice from Kinesis, though this relieves the memory pressure. It looks like, based on the Kinesis, unlike Kafka, we have to read the data first to get the offset range, so caching RDD is inevitable if we want good performance. So we can keep the latest changes.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
+ if (fullyConsumed) {
+ log.info("Expired shard {} was fully consumed (lastSeq >= endSeq); pruning from checkpoint",
+ shardId);
+ } else {
+ if (failOnDataLoss) {
+ throw new HoodieReadFromSourceException("Checkpoint references expired shard " + shardId
+ + " with unread data (lastSeq < endSeq or no endSeq stored). Data loss may have occurred. "
+ + "Set " + KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS.key() + "=false to continue.");
+ }
+ log.warn("Expired shard {} may have unread data; pruning and continuing (failOnDataLoss=false)",
+ shardId);
+ }
+ }
+ }
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId));
+ if (lastSeq != null && !lastSeq.isEmpty()) {
+ fromSequenceNumbers.put(shardId, lastSeq);
+ }
+ }
+ }
+ }
+ }
+
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = (shard.sequenceNumberRange() != null
+ && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? Option.of(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+
+ int targetParallelism = minPartitions > 0
+ ? (int) Math.max(minPartitions, ranges.size())
+ : ranges.size();
+ metrics.updateStreamerSourceParallelism(targetParallelism);
+
+ log.info("About to read up to {} events from {} shards in stream {} (target parallelism: {})",
+ numEvents, ranges.size(), streamName, targetParallelism);
+ return ranges.toArray(new KinesisShardRange[0]);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded for stream " + streamName, e);
+ } catch (InvalidArgumentException e) {
+ throw new HoodieReadFromSourceException("Invalid Kinesis request: " + e.getMessage(), e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded: " + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Result of reading from a shard: records and the last sequence number for checkpoint.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class ShardReadResult implements java.io.Serializable {
+ private final List<Record> records;
+ private final Option<String> lastSequenceNumber;
+ /** True when nextShardIterator was null, meaning the shard has no more records to return. */
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Read records from a single shard.
+ * @param enableDeaggregation when true, de-aggregates KPL records into individual user records
+ */
+ public static ShardReadResult readShardRecords(KinesisClient client, String streamName,
+ KinesisShardRange range, KinesisSourceConfig.KinesisStartingPosition defaultPosition,
+ int maxRecordsPerRequest, long intervalMs, long maxTotalRecords,
+ boolean enableDeaggregation) throws InterruptedException {
+ String shardIterator;
+ try {
+ shardIterator = getShardIterator(client, streamName, range, defaultPosition);
+ } catch (InvalidArgumentException e) {
+ // GetShardIterator throws InvalidArgumentException (not ExpiredIteratorException) when the
+ // requested sequence number is past the stream's retention window.
+ throw new HoodieReadFromSourceException("Sequence number in checkpoint is expired or invalid for shard "
+ + range.getShardId() + ". Reset the checkpoint to recover.", e);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Shard or stream not found: " + range.getShardId(), e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+ List<Record> allRecords = new ArrayList<>();
+ String lastSequenceNumber = null;
+ int requestCount = 0;
+
+ while (allRecords.size() < maxTotalRecords && shardIterator != null) {
+ GetRecordsResponse response;
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIterator)
+ .limit(Math.min(maxRecordsPerRequest, (int) (maxTotalRecords - allRecords.size())))
+ .build());
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", range.getShardId());
+ break;
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded reading shard " + range.getShardId(), e);
+ }
+
+ List<Record> records = response.records();
+ // Update shardIterator before the empty check so its null-ness correctly reflects end-of-shard
+ // even when the final response carries 0 records (closed shard fully exhausted).
+ shardIterator = response.nextShardIterator();
+ // CASE 1: No records returned: stop polling. nextShardIterator can be non-null when at LATEST with no new
+ // data; continuing would cause an infinite loop of empty GetRecords calls.
+ if (records.isEmpty()) {
+ break;
+ }
+ // CASE 2: records returned.
+ List<Record> toAdd = enableDeaggregation ? KinesisDeaggregator.deaggregate(records) : records;
+ for (Record r : toAdd) {
+ allRecords.add(r);
+ }
+ // Checkpoint uses the last Kinesis record's sequence number (from raw records, not deaggregated)
+ lastSequenceNumber = records.get(records.size() - 1).sequenceNumber();
+
+ requestCount++;
+ // This is for rate limiting
+ if (shardIterator != null && intervalMs > 0) {
+ Thread.sleep(intervalMs);
+ }
+ }
Review Comment:
Sg
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -210,7 +218,8 @@ private static String recordToJsonStatic(Record record, String shardId, boolean
}
return OBJECT_MAPPER.writeValueAsString(node);
} catch (Exception e) {
- return dataStr;
+ // We can disable the flag for mitigation.
+ throw new RuntimeException("Failed to add metadata fields", e);
Review Comment:
Use `HoodieException`
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
Review Comment:
```suggestion
.key(PREFIX + "partitions")
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
.markAdvanced()
.withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
.key(PREFIX + "enable.deaggregation")
.defaultValue(true)
- .withAlternatives(OLD_PREFIX + "enable.deaggregation")
.markAdvanced()
.withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
- .key(PREFIX + "enable.failOnDataLoss")
+ .key(PREFIX + "enable.fail.on.data.loss")
.defaultValue(false)
- .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
.markAdvanced()
- .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
.key(PREFIX + "starting.position")
.defaultValue("LATEST")
- .withAlternatives(OLD_PREFIX + "starting.position")
.markAdvanced()
- .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
- public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
- .key(PREFIX + "getRecords.maxRecords")
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
.defaultValue(10000)
- .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
.markAdvanced()
.withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
- .key(PREFIX + "getRecords.intervalMs")
+ .key(PREFIX + "get_records.interval.ms")
Review Comment:
```suggestion
.key(PREFIX + "get.records.interval.ms")
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
Review Comment:
Should this be `false` by default?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+
+import javax.annotation.concurrent.Immutable;
+
+import static org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
+/**
+ * Kinesis Source Configs for Hudi Streamer.
+ */
+@Immutable
+@ConfigClassProperty(name = "Kinesis Source Configs",
+ groupName = ConfigGroups.Names.HUDI_STREAMER,
+ subGroupName = ConfigGroups.SubGroupNames.DELTA_STREAMER_SOURCE,
+ description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
+public class KinesisSourceConfig extends HoodieConfig {
+
+ public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
+
+ private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
+ private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
+
+ public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
+ .key(PREFIX + "stream.name")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "stream.name")
+ .withDocumentation("Kinesis Data Streams stream name.");
+
+ public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
+ .key(PREFIX + "region")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "region")
+ .withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
+
+ public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
+ .key(PREFIX + "endpoint.url")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "endpoint.url")
+ .markAdvanced()
+ .withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ + "If not set, uses the default AWS endpoint for the region.");
+
+ public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
+ .key(PREFIX + "access.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "access.key")
+ .markAdvanced()
+ .withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
+ .key(PREFIX + "secret.key")
+ .noDefaultValue()
+ .withAlternatives(OLD_PREFIX + "secret.key")
+ .markAdvanced()
+ .withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ + "If not set with endpoint, uses the default AWS credential chain.");
+
+ public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
+ .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .defaultValue(5000000L)
+ .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
+
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
+ .key(PREFIX + "minPartitions")
+ .defaultValue(0L)
+ .withAlternatives(OLD_PREFIX + "minPartitions")
+ .markAdvanced()
+ .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ + "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
+ + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
+ + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+
+ public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
+ .key(PREFIX + "append.offsets")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "append.offsets")
+ .markAdvanced()
+ .withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
+
+ public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
+ .key(PREFIX + "enable.deaggregation")
+ .defaultValue(true)
+ .withAlternatives(OLD_PREFIX + "enable.deaggregation")
+ .markAdvanced()
+ .withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ + "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
+
+ public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
+ .key(PREFIX + "enable.failOnDataLoss")
+ .defaultValue(false)
+ .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
+ .markAdvanced()
+ .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+
+ public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
+ .key(PREFIX + "starting.position")
+ .defaultValue("LATEST")
+ .withAlternatives(OLD_PREFIX + "starting.position")
+ .markAdvanced()
+ .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+
+ public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
+ .key(PREFIX + "getRecords.maxRecords")
+ .defaultValue(10000)
+ .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
+ .markAdvanced()
+ .withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
Review Comment:
The docs mention that "Kinesis limit is 10000". Thus asking about the validation.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
Review Comment:
Add `.sinceVersion("1.2.0")`
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
.markAdvanced()
.withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
.key(PREFIX + "enable.deaggregation")
.defaultValue(true)
- .withAlternatives(OLD_PREFIX + "enable.deaggregation")
.markAdvanced()
.withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
- .key(PREFIX + "enable.failOnDataLoss")
+ .key(PREFIX + "enable.fail.on.data.loss")
.defaultValue(false)
- .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
.markAdvanced()
- .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
.key(PREFIX + "starting.position")
.defaultValue("LATEST")
- .withAlternatives(OLD_PREFIX + "starting.position")
.markAdvanced()
- .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
- public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
- .key(PREFIX + "getRecords.maxRecords")
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
.defaultValue(10000)
- .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
.markAdvanced()
.withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
- .key(PREFIX + "getRecords.intervalMs")
+ .key(PREFIX + "get_records.interval.ms")
.defaultValue(200L)
- .withAlternatives(OLD_PREFIX + "getRecords.intervalMs")
.markAdvanced()
.withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
+ public static final ConfigProperty<Boolean> KINESIS_PERSIST_FETCH_RDD = ConfigProperty
+ .key(PREFIX + "persist.fetch.rdd")
+ .defaultValue(false)
+ .markAdvanced()
+ .withDocumentation("When enabled, the fetch RDD is persisted (MEMORY_AND_DISK) so it can be used "
+ + "both for record RDD and checkpoint summaries without recomputation. When false, the fetch "
+ + "is computed twice (once for checkpoint, once when the record RDD is consumed), which can "
+ + "cause duplicate records to be written. Set to true for correct, duplicate-free ingestion.");
+
Review Comment:
If setting to `false` does not bring benefit, let's remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034690650
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
Review Comment:
ok. Will add to a follow up ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185544485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324115011
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.71%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|4|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.71% -21.19%
+ Complexity 28537 16613 -11924
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 54509 -39813
- Misses 34996 54018 +19022
+ Partials 7592 5719 -1873
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.99% <0.00%> (-17.48%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324123789
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 54.20%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 54.20% -14.70%
+ Complexity 28537 19178 -9359
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 61925 -32397
- Misses 34996 46672 +11676
+ Partials 7592 5649 -1943
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.17% <0.00%> (-4.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1289 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324124537
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.72%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.72% -9.18%
+ Complexity 28537 21120 -7417
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 68229 -26093
- Misses 34996 39832 +4836
+ Partials 7592 6185 -1407
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.17% <0.00%> (-4.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <0.00%> (-0.86%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1117 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324670970
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.43%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 23 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|3|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 45.43% -23.47%
+ Complexity 28537 15527 -13010
=============================================
Files 2480 2009 -471
Lines 136910 111811 -25099
Branches 16679 14071 -2608
=============================================
- Hits 94322 50799 -43523
- Misses 34996 55632 +20636
+ Partials 7592 5380 -2212
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.47% <0.00%> (-12.81%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1603 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324692996
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.57%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.57% -21.32%
+ Complexity 28537 16578 -11959
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 54354 -39968
- Misses 34996 54155 +19159
+ Partials 7592 5737 -1855
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.10% <0.00%> (-16.38%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324703270
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.67%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.67% -21.22%
+ Complexity 28537 16609 -11928
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 54472 -39850
- Misses 34996 54050 +19054
+ Partials 7592 5724 -1868
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.39% <0.00%> (-17.08%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1546 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324743772
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.24%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 56.24% -12.66%
+ Complexity 28537 19305 -9232
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 64255 -30067
- Misses 34996 43632 +8636
+ Partials 7592 6359 -1233
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.17% <0.00%> (-9.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1320 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324837033
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.44%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.44% -2.45%
+ Complexity 28537 23178 -5359
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 75910 -18412
+ Misses 34996 31574 -3422
+ Partials 7592 6762 -830
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.59% <0.00%> (-0.88%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [853 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325631811
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.44%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.44% -9.46%
+ Complexity 28537 20918 -7619
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 69489 -24833
- Misses 34996 40307 +5311
+ Partials 7592 7109 -483
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.03% <0.00%> (-17.44%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.45% <0.00%> (-0.83%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1305 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325610673
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.35%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|5|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.35% -21.55%
+ Complexity 28537 16860 -11677
=============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
=============================================
- Hits 94322 55355 -38967
- Misses 34996 55542 +20546
+ Partials 7592 6008 -1584
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.03% <0.00%> (-17.44%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.65% <0.00%> (-12.63%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1572 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325573006
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.14%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 23 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|2|
>|spark-java-tests|14|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.14% -20.75%
+ Complexity 28537 16398 -12139
=============================================
Files 2480 2008 -472
Lines 136910 112038 -24872
Branches 16679 14144 -2535
=============================================
- Hits 94322 53939 -40383
- Misses 34996 52425 +17429
+ Partials 7592 5674 -1918
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.16% <0.00%> (-16.32%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.86% <0.00%> (-12.42%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1593 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325679431
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 62.55%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 10 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|7|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 62.55% -6.34%
+ Complexity 28537 22027 -6510
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 73130 -21192
- Misses 34996 36645 +1649
+ Partials 7592 7130 -462
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.81% <0.00%> (-7.66%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.91% <0.00%> (-0.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1137 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325861995
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.81%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.81% -3.09%
+ Complexity 28537 23512 -5025
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76939 -17383
+ Misses 34996 32961 -2035
+ Partials 7592 7005 -587
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.64% <0.00%> (-0.84%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [860 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327461599
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `70.08130%` with `184 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.96%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cb751bb`](https://app.codecov.io/gh/apache/hudi/commit/cb751bb9a7be2db62364d07202a29ca1d17f863b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [55 Missing and 30 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 71.62% | [28 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cb751bb). Click for more details.
>
> <details><summary>HEAD has 11 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cb751bb) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.96% -8.93%
+ Complexity 28537 21157 -7380
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 70102 -24220
- Misses 34996 39586 +4590
+ Partials 7592 7217 -375
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.83% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.59% <0.00%> (-17.88%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.40% <0.00%> (-0.88%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <70.08%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `71.62% <71.62%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1279 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327584125
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* cb751bb9a7be2db62364d07202a29ca1d17f863b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648)
* 7af0201256fc80be36c78bc2e56df221b876902a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327792307
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.94%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7af0201`](https://app.codecov.io/gh/apache/hudi/commit/7af0201256fc80be36c78bc2e56df221b876902a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (7af0201). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (7af0201) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 53.94% -14.95%
+ Complexity 28537 12397 -16140
=============================================
Files 2480 1433 -1047
Lines 136910 71927 -64983
Branches 16679 8203 -8476
=============================================
- Hits 94322 38802 -55520
+ Misses 34996 29650 -5346
+ Partials 7592 3475 -4117
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.02%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1829 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328034492
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
}, {
"hash" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb",
"triggerType" : "PUSH"
}, {
"hash" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13648",
"triggerID" : "cb751bb9a7be2db62364d07202a29ca1d17f863b",
"triggerType" : "PUSH"
}, {
"hash" : "7af0201256fc80be36c78bc2e56df221b876902a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650",
"triggerID" : "7af0201256fc80be36c78bc2e56df221b876902a",
"triggerType" : "PUSH"
}, {
"hash" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13652",
"triggerID" : "a193e7e13b56a25526e18c3eec95cbdf3d16c79c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* 937bbb2851bad8a3a42c7a7f8cde9cd6135e4bcb UNKNOWN
* 7af0201256fc80be36c78bc2e56df221b876902a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13650)
* a193e7e13b56a25526e18c3eec95cbdf3d16c79c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13652)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329018904
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.63%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|4|
>|spark-java-tests|14|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 48.63% -20.27%
+ Complexity 28537 16576 -11961
=============================================
Files 2480 2009 -471
Lines 136910 112020 -24890
Branches 16679 14154 -2525
=============================================
- Hits 94322 54477 -39845
- Misses 34996 51778 +16782
+ Partials 7592 5765 -1827
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.74% <0.00%> (-16.73%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.92% <0.00%> (-12.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1563 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329128601
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.01%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.01% -0.88%
- Complexity 28537 28884 +347
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95586 +1264
- Misses 34996 37111 +2115
- Partials 7592 7841 +249
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.36% <0.00%> (-1.11%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [95 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329147123
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.03%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.03% -0.86%
- Complexity 28537 28893 +356
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95613 +1291
- Misses 34996 37075 +2079
- Partials 7592 7850 +258
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.52% <0.00%> (-0.95%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [86 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937785661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a517e2685c28a1101af008e73abfe04522484a17 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166)
* 82ade47dcdb28b4fd8db26ff39980c5e72fca27b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937936958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 01a34fe2172609ecab7c865f2e3052bc82e6cc9c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat:Create KinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3930885356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* fec6001976389d10a38feda0a413fb6043f23ace Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134)
* d09affde11485fedf2aba0f93230da54669bf307 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3937863492
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 82ade47dcdb28b4fd8db26ff39980c5e72fca27b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172)
* 01a34fe2172609ecab7c865f2e3052bc82e6cc9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3037295633
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisDeaggregator;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddMetaFields;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddMetaFields = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ // STEP 1: Collect all available shards for the stream: open/closed shards.
+ KinesisOffsetGen.KinesisShardRange[] allOpenClosedShardRanges = offsetGen.getNextShardRanges(lastCheckpoint, sourceLimit);
+ // STEP 2: Filter out shards with no unread records to avoid unnecessary GetRecords calls.
+ boolean useLatestStartingPositionStrategy =
+ offsetGen.getStartingPositionStrategy() == KinesisSourceConfig.KinesisStartingPositionStrategy.LATEST;
+ int numShardsBeforeFilter = allOpenClosedShardRanges.length;
+ KinesisOffsetGen.KinesisShardRange[] shardRangesWithUnreadRecords = Arrays.stream(allOpenClosedShardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestStartingPositionStrategy))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (numShardsBeforeFilter > shardRangesWithUnreadRecords.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ numShardsBeforeFilter - shardRangesWithUnreadRecords.length, shardRangesWithUnreadRecords.length);
+ }
+ // When nothing to read, return empty batch and previous checkpoint if any.
+ if (shardRangesWithUnreadRecords.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
+ }
+ // STEP 3: Otherwise, do the read.
+ T batch = toBatch(shardRangesWithUnreadRecords, sourceLimit);
+ // STEP 4: Generate checkpoint.
+ // Pass allOpenClosedShardRanges so filtered-out shards are preserved in the checkpoint; otherwise
+ // next run would re-read them from TRIM_HORIZON and cause duplicates
+ String checkpointStr = createCheckpointFromBatch(batch, shardRangesWithUnreadRecords, allOpenClosedShardRanges);
+ // STEP 5: Emit metrics.
+ long totalMsgs = getRecordCount(batch);
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, totalMsgs);
+ log.info("Read {} records from Kinesis stream {} with {} shards, checkpoint: {}",
+ totalMsgs, offsetGen.getStreamName(), shardRangesWithUnreadRecords.length, checkpointStr);
+
+ return new InputBatch<>(Option.of(batch), checkpointStr);
+ }
+
+ /** Upper bound on consecutive empty GetRecords responses before giving up on a shard. */
+ private static final int MAX_EMPTY_RESPONSES_FROM_GET_RECORDS = 100;
+
+ /**
+ * Lazy iterator over records from a single Kinesis shard.
+ *
+ * <p>Records are fetched one GetRecords page at a time; the next page is only requested once all
+ * records from the current page have been consumed. This avoids holding the full shard batch in
+ * executor memory simultaneously with the caller's output collection.
+ *
+ * <p>After {@link #hasNext()} returns {@code false} callers must read
+ * {@link #getLastSequenceNumber()} and {@link #isReachedEndOfShard()} to obtain checkpoint state.
+ *
+ * <p><b>lastSequenceNumber correctness invariant:</b> the sequence number is taken from the last
+ * <em>raw</em> Kinesis record (pre-deaggregation) of a page and is only committed once all
+ * deaggregated records from that page have been yielded. This guarantees the checkpoint never
+ * advances past records that have not yet been returned to the caller.
+ */
+ public static class ShardRecordIterator implements Iterator<Record> {
+ private final KinesisClient client;
+ private final String shardId;
+ private final int maxRecordsPerRequest;
+ private final long requestIntervalMs;
+ private final long maxTotalRecords;
+ private final boolean enableDeaggregation;
+ private final long retryInitialIntervalMs;
+ private final long retryMaxIntervalMs;
+ private final long throttleTimeoutMs;
+
+ /** Current position in the Kinesis shard; null means the shard is exhausted. */
+ private String shardIteratorStr;
+ /** Records from the most recently fetched page, ready to be yielded. */
+ private Iterator<Record> currentPage = Collections.emptyIterator();
+ /**
+ * Raw lastSeq of the page currently being consumed. Moved to {@link #lastSequenceNumber} only
+ * when the page iterator is fully exhausted, ensuring the checkpoint never skips records.
+ */
+ private String pendingPageLastSeq = null;
+ /** Checkpoint-safe lastSeq: reflects only fully-consumed pages. */
+ private String lastSequenceNumber = null;
+ private boolean reachedEndOfShard = false;
+ /** True once no further GetRecords calls should be made. */
+ private boolean fetchingDone = false;
+ private long totalConsumed = 0;
+ private int emptyPageCount = 0;
+
+ /**
+ * Dynamically tuned records-per-request limit.
+ * Halved on each ProvisionedThroughputExceededException and held there for the rest of the shard read.
+ */
+ private int currentMaxRecords;
+ /** Epoch ms of the last successful GetRecords call; used to enforce {@link #throttleTimeoutMs}. */
+ private long lastSuccessTimeMs;
+
+ public ShardRecordIterator(String initialShardIterator, KinesisClient client, String shardId,
+ int maxRecordsPerRequest, long requestIntervalMs, long maxTotalRecords, boolean enableDeaggregation,
+ long retryInitialIntervalMs, long retryMaxIntervalMs, long throttleTimeoutMs) {
+ this.shardIteratorStr = initialShardIterator;
+ this.client = client;
+ this.shardId = shardId;
+ this.maxRecordsPerRequest = maxRecordsPerRequest;
+ this.requestIntervalMs = requestIntervalMs;
+ this.maxTotalRecords = maxTotalRecords;
+ this.enableDeaggregation = enableDeaggregation;
+ this.retryInitialIntervalMs = retryInitialIntervalMs;
+ this.retryMaxIntervalMs = retryMaxIntervalMs;
+ this.throttleTimeoutMs = throttleTimeoutMs;
+ this.currentMaxRecords = maxRecordsPerRequest;
+ this.lastSuccessTimeMs = System.currentTimeMillis();
+ }
+
+ @Override
+ public boolean hasNext() {
+ while (true) {
+ if (currentPage.hasNext()) {
+ return true;
+ }
+ // Current page fully consumed: commit its lastSeq before moving on.
+ commitPendingPageLastSeq();
+ if (fetchingDone) {
+ return false;
+ }
+ fetchNextPage();
+ // Loop: if the page was empty, try fetching again (up to MAX_EMPTY_RESPONSES limit).
+ }
+ }
+
+ @Override
+ public Record next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException("No more records for shard " + shardId);
+ }
+ totalConsumed++;
+ return currentPage.next();
Review Comment:
_⚠️ Potential issue_ | _🟠 Major_
**Deaggregation can blow past the configured batch cap.**
`totalConsumed` is only consulted when deciding whether to fetch another page. Once Line 281 expands a raw Kinesis page into many user records, Lines 191-210 will drain that whole page before the quota is checked again. That means `hoodie.streamer.source.kinesis.max.events` stops being a real upper bound as soon as KPL aggregation is enabled.
Also applies to: 278-283
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java`
around lines 191 - 210, The iterator drains an entire deaggregated currentPage
in next() without rechecking the totalConsumed quota, allowing deaggregation to
exceed hoodie.streamer.source.kinesis.max.events; modify hasNext() and next() so
they both consult totalConsumed against the configured max (e.g., maxEvents)
before yielding a record: when currentPage has elements, do not simply return
true/unconditionally next() — instead compute remainingAllowed = maxEvents -
totalConsumed and if remainingAllowed <= 0 treat as no more records; if
currentPage contains more elements than remainingAllowed, only yield up to
remainingAllowed (by limiting the per-page iterator or wrapping
currentPage.next() to stop after remainingAllowed) and ensure
commitPendingPageLastSeq()/fetchNextPage() behavior still runs when a page is
fully consumed. Update logic in hasNext(), next(), and any loop that calls
fetchNextPage()/commitPendingPageLastSeq() to enforce this quota check so
deaggregation cannot blow past the configured cap (references: hasNext(),
next(), commitPendingPageLastSeq(), fetchNextPage(), totalConsumed).
```
</details>
<!-- fingerprinting:phantom:medusa:grasshopper:2d6277d7-75f4-4e28-b60a-1efa18b71895 -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410525)) (source:comment#3036410525)
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJsonKinesisSource.java:
##########
@@ -0,0 +1,480 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.lang.reflect.Method;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.utilities.config.KinesisSourceConfig.KINESIS_REGION;
+import static org.apache.hudi.utilities.config.KinesisSourceConfig.KINESIS_STARTING_POSITION;
+import static org.apache.hudi.utilities.config.KinesisSourceConfig.KINESIS_STREAM_NAME;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+
+/**
+ * Unit tests for JsonKinesisSource.
+ */
+class TestJsonKinesisSource extends SparkClientFunctionalTestHarness {
+
+ private static final String STREAM_NAME = "test-stream";
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private TestableJsonKinesisSource source;
+
+ @BeforeEach
+ void setup() {
+ TypedProperties props = new TypedProperties();
+ props.setProperty(KINESIS_STREAM_NAME.key(), STREAM_NAME);
+ props.setProperty(KINESIS_REGION.key(), "us-east-1");
+ props.setProperty(KINESIS_STARTING_POSITION.key(), "TRIM_HORIZON");
+
+ source = new TestableJsonKinesisSource(
+ props, jsc(), spark(), null, mock(HoodieIngestionMetrics.class));
+ }
+
+ // --- recordToJsonStatic tests ---
+
+ private static String recordToJson(Record record, String shardId, boolean shouldAddOffsets)
+ throws Exception {
+ Method m = JsonKinesisSource.class.getDeclaredMethod(
+ "recordToJsonStatic", Record.class, String.class, boolean.class);
+ m.setAccessible(true);
+ return (String) m.invoke(null, record, shardId, shouldAddOffsets);
+ }
+
+ private static Record kinesisRecord(String data, String partitionKey, String sequenceNumber,
+ Instant approximateArrivalTimestamp) {
+ Record.Builder builder = Record.builder()
+ .data(SdkBytes.fromUtf8String(data))
+ .partitionKey(partitionKey)
+ .sequenceNumber(sequenceNumber);
+ if (approximateArrivalTimestamp != null) {
+ builder.approximateArrivalTimestamp(approximateArrivalTimestamp);
+ }
+ return builder.build();
+ }
+
+ @Test
+ void testRecordToJsonPassThroughWhenShouldAddOffsetsFalse() throws Exception {
+ String json = "{\"id\":1,\"name\":\"alice\"}";
+ Record record = kinesisRecord(json, "pk1", "49590382471490958861609854428592832524486083118",
+ Instant.ofEpochMilli(1700000000000L));
+
+ String result = recordToJson(record, "shardId-000000000000", false);
+
+ assertEquals(json, result);
+ }
+
+ @Test
+ void testRecordToJsonAddsOffsetFieldsWhenShouldAddOffsetsTrue() throws Exception {
+ String json = "{\"id\":1,\"name\":\"alice\"}";
+ Record record = kinesisRecord(json, "pk1", "49590382471490958861609854428592832524486083118",
+ Instant.ofEpochMilli(1700000000000L));
+
+ String result = recordToJson(record, "shardId-000000000001", true);
+
+ JsonNode node = MAPPER.readTree(result);
+ assertEquals("49590382471490958861609854428592832524486083118",
+ node.get("_hoodie_kinesis_source_sequence_number").asText());
+ assertEquals("shardId-000000000001", node.get("_hoodie_kinesis_source_shard_id").asText());
+ assertEquals("pk1", node.get("_hoodie_kinesis_source_partition_key").asText());
+ assertEquals(1700000000000L, node.get("_hoodie_kinesis_source_timestamp").asLong());
+ assertEquals(1, node.get("id").asInt());
+ assertEquals("alice", node.get("name").asText());
+ }
+
+ @Test
+ void testRecordToJsonEmptyStringReturnsNull() throws Exception {
+ Record record = kinesisRecord("", "pk1", "seq123", Instant.now());
+
+ String result = recordToJson(record, "shardId-0", false);
+
+ assertNull(result);
+ }
+
+ @Test
+ void testRecordToJsonWhitespaceOnlyReturnsNull() throws Exception {
+ Record record = kinesisRecord(" ", "pk1", "seq123", Instant.now());
+
+ String result = recordToJson(record, "shardId-0", false);
+
+ assertNull(result);
+ }
+
+ @Test
+ void testRecordToJsonNullTimestampNotAdded() throws Exception {
+ String json = "{\"id\":1}";
+ Record record = kinesisRecord(json, "pk1", "seq123", null);
+
+ String result = recordToJson(record, "shardId-0", true);
+
+ JsonNode node = MAPPER.readTree(result);
+ assertTrue(node.has("_hoodie_kinesis_source_sequence_number"));
+ assertTrue(node.has("_hoodie_kinesis_source_shard_id"));
+ assertTrue(node.has("_hoodie_kinesis_source_partition_key"));
+ assertEquals(false, node.has("_hoodie_kinesis_source_timestamp"));
+ }
+
+ @Test
+ void testRecordToJsonInvalidJsonWithShouldAddOffsetsReturnsOriginalString() throws Exception {
+ String invalidJson = "not valid json {";
+ Record record = kinesisRecord(invalidJson, "pk1", "seq123", Instant.now());
+
+ String result = recordToJson(record, "shardId-0", true);
+
+ assertEquals(invalidJson, result);
+ }
Review Comment:
<a href="#"><img alt="P0" src="https://greptile-static-assets.s3.amazonaws.com/badges/p0.svg?v=7" align="top"></a> **Test asserts wrong behavior — will fail at runtime**
The test name says "ReturnsOriginalString", but the production code in `JsonKinesisSource.recordToJsonStatic` does the opposite: when `shouldAddMetaFields=true` and `OBJECT_MAPPER.readTree(dataStr)` throws a `JsonParseException` for invalid JSON, the `catch` block rethrows as `HoodieException("Failed to add metadata fields", e)`. That exception propagates through reflection as an `InvocationTargetException`, so `assertEquals(invalidJson, result)` is never reached.
The test either needs to assert that an exception is thrown:
```java
// Option A – assert exception
assertThrows(HoodieException.class,
() -> recordToJson(record, "shardId-0", true));
```
or the production code must be changed to silently fall back (which would conflict with the explicit `throw` in the `catch` block and the comment "We can disable the flag for mitigation").
— *Greptile* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036407228)) (source:comment#3036407228)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisDeaggregator;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddMetaFields;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddMetaFields = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ // STEP 1: Collect all available shards for the stream: open/closed shards.
+ KinesisOffsetGen.KinesisShardRange[] allOpenClosedShardRanges = offsetGen.getNextShardRanges(lastCheckpoint, sourceLimit);
+ // STEP 2: Filter out shards with no unread records to avoid unnecessary GetRecords calls.
+ boolean useLatestStartingPositionStrategy =
+ offsetGen.getStartingPositionStrategy() == KinesisSourceConfig.KinesisStartingPositionStrategy.LATEST;
+ int numShardsBeforeFilter = allOpenClosedShardRanges.length;
+ KinesisOffsetGen.KinesisShardRange[] shardRangesWithUnreadRecords = Arrays.stream(allOpenClosedShardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestStartingPositionStrategy))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (numShardsBeforeFilter > shardRangesWithUnreadRecords.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ numShardsBeforeFilter - shardRangesWithUnreadRecords.length, shardRangesWithUnreadRecords.length);
+ }
+ // When nothing to read, return empty batch and previous checkpoint if any.
+ if (shardRangesWithUnreadRecords.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
+ }
+ // STEP 3: Otherwise, do the read.
+ T batch = toBatch(shardRangesWithUnreadRecords, sourceLimit);
+ // STEP 4: Generate checkpoint.
+ // Pass allOpenClosedShardRanges so filtered-out shards are preserved in the checkpoint; otherwise
+ // next run would re-read them from TRIM_HORIZON and cause duplicates
+ String checkpointStr = createCheckpointFromBatch(batch, shardRangesWithUnreadRecords, allOpenClosedShardRanges);
+ // STEP 5: Emit metrics.
+ long totalMsgs = getRecordCount(batch);
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, totalMsgs);
+ log.info("Read {} records from Kinesis stream {} with {} shards, checkpoint: {}",
+ totalMsgs, offsetGen.getStreamName(), shardRangesWithUnreadRecords.length, checkpointStr);
+
+ return new InputBatch<>(Option.of(batch), checkpointStr);
+ }
+
+ /** Upper bound on consecutive empty GetRecords responses before giving up on a shard. */
+ private static final int MAX_EMPTY_RESPONSES_FROM_GET_RECORDS = 100;
+
+ /**
+ * Lazy iterator over records from a single Kinesis shard.
+ *
+ * <p>Records are fetched one GetRecords page at a time; the next page is only requested once all
+ * records from the current page have been consumed. This avoids holding the full shard batch in
+ * executor memory simultaneously with the caller's output collection.
+ *
+ * <p>After {@link #hasNext()} returns {@code false} callers must read
+ * {@link #getLastSequenceNumber()} and {@link #isReachedEndOfShard()} to obtain checkpoint state.
+ *
+ * <p><b>lastSequenceNumber correctness invariant:</b> the sequence number is taken from the last
+ * <em>raw</em> Kinesis record (pre-deaggregation) of a page and is only committed once all
+ * deaggregated records from that page have been yielded. This guarantees the checkpoint never
+ * advances past records that have not yet been returned to the caller.
+ */
+ public static class ShardRecordIterator implements Iterator<Record> {
+ private final KinesisClient client;
+ private final String shardId;
+ private final int maxRecordsPerRequest;
+ private final long requestIntervalMs;
+ private final long maxTotalRecords;
+ private final boolean enableDeaggregation;
+ private final long retryInitialIntervalMs;
+ private final long retryMaxIntervalMs;
+ private final long throttleTimeoutMs;
+
+ /** Current position in the Kinesis shard; null means the shard is exhausted. */
+ private String shardIteratorStr;
+ /** Records from the most recently fetched page, ready to be yielded. */
+ private Iterator<Record> currentPage = Collections.emptyIterator();
+ /**
+ * Raw lastSeq of the page currently being consumed. Moved to {@link #lastSequenceNumber} only
+ * when the page iterator is fully exhausted, ensuring the checkpoint never skips records.
+ */
+ private String pendingPageLastSeq = null;
+ /** Checkpoint-safe lastSeq: reflects only fully-consumed pages. */
+ private String lastSequenceNumber = null;
+ private boolean reachedEndOfShard = false;
+ /** True once no further GetRecords calls should be made. */
+ private boolean fetchingDone = false;
+ private long totalConsumed = 0;
+ private int emptyPageCount = 0;
+
+ /**
+ * Dynamically tuned records-per-request limit.
+ * Halved on each ProvisionedThroughputExceededException and held there for the rest of the shard read.
+ */
+ private int currentMaxRecords;
+ /** Epoch ms of the last successful GetRecords call; used to enforce {@link #throttleTimeoutMs}. */
+ private long lastSuccessTimeMs;
+
+ public ShardRecordIterator(String initialShardIterator, KinesisClient client, String shardId,
+ int maxRecordsPerRequest, long requestIntervalMs, long maxTotalRecords, boolean enableDeaggregation,
+ long retryInitialIntervalMs, long retryMaxIntervalMs, long throttleTimeoutMs) {
+ this.shardIteratorStr = initialShardIterator;
+ this.client = client;
+ this.shardId = shardId;
+ this.maxRecordsPerRequest = maxRecordsPerRequest;
+ this.requestIntervalMs = requestIntervalMs;
+ this.maxTotalRecords = maxTotalRecords;
+ this.enableDeaggregation = enableDeaggregation;
+ this.retryInitialIntervalMs = retryInitialIntervalMs;
+ this.retryMaxIntervalMs = retryMaxIntervalMs;
+ this.throttleTimeoutMs = throttleTimeoutMs;
+ this.currentMaxRecords = maxRecordsPerRequest;
+ this.lastSuccessTimeMs = System.currentTimeMillis();
+ }
+
+ @Override
+ public boolean hasNext() {
+ while (true) {
+ if (currentPage.hasNext()) {
+ return true;
+ }
+ // Current page fully consumed: commit its lastSeq before moving on.
+ commitPendingPageLastSeq();
+ if (fetchingDone) {
+ return false;
+ }
+ fetchNextPage();
+ // Loop: if the page was empty, try fetching again (up to MAX_EMPTY_RESPONSES limit).
+ }
+ }
+
+ @Override
+ public Record next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException("No more records for shard " + shardId);
+ }
+ totalConsumed++;
+ return currentPage.next();
+ }
+
+ public Option<String> getLastSequenceNumber() {
+ return Option.ofNullable(lastSequenceNumber);
+ }
+
+ public boolean isReachedEndOfShard() {
+ return reachedEndOfShard;
+ }
+
+ private void commitPendingPageLastSeq() {
+ if (pendingPageLastSeq != null) {
+ lastSequenceNumber = pendingPageLastSeq;
+ pendingPageLastSeq = null;
+ }
+ }
+
+ private void fetchNextPage() {
+ if (shardIteratorStr == null || totalConsumed >= maxTotalRecords) {
+ fetchingDone = true;
+ return;
+ }
+ GetRecordsResponse response;
+ int attempt = 0;
+ while (true) {
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIteratorStr)
+ .limit(Math.min(currentMaxRecords, (int) (maxTotalRecords - totalConsumed)))
+ .build());
+ lastSuccessTimeMs = System.currentTimeMillis();
+ break;
+ } catch (ExpiredIteratorException e) {
+ log.warn("Shard iterator expired for {} during GetRecords, stopping read", shardId);
+ fetchingDone = true;
+ return;
+ } catch (ProvisionedThroughputExceededException e) {
+ long nowMs = System.currentTimeMillis();
+ if (nowMs - lastSuccessTimeMs > throttleTimeoutMs) {
+ throw new HoodieReadFromSourceException(
+ "Kinesis throughput exceeded for shard " + shardId + ": no successful fetch within "
+ + throttleTimeoutMs + " ms. Last successful fetch, or first fetch was " + (nowMs - lastSuccessTimeMs) + " ms ago.", e);
+ }
+ // Halve the per-request limit to reduce pressure; floor at 1.
+ int prevLimit = currentMaxRecords;
+ currentMaxRecords = Math.max(1, currentMaxRecords / 2);
+ // Use attempt count only to compute exponential backoff delay, not as a stop condition.
+ long waitMs = Math.min(retryInitialIntervalMs * (1L << Math.min(attempt, 30)), retryMaxIntervalMs);
+ waitMs += ThreadLocalRandom.current().nextInt(500);
+ log.warn("Throughput exceeded for shard {}: halving records/request from {} to {}, retry after {} ms "
+ + "(no success for {} ms, will give up after {} ms)",
+ shardId, prevLimit, currentMaxRecords, waitMs, nowMs - lastSuccessTimeMs, throttleTimeoutMs);
+ try {
+ Thread.sleep(waitMs);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ throw new HoodieReadFromSourceException("Interrupted while backing off for shard " + shardId, ie);
+ }
+ attempt++;
Review Comment:
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=7" align="top"></a> **Lexicographic sequence number comparison may be incorrect for different-width values**
`lastSeq.compareTo(endSeq)` is used to detect fully-consumed closed shards. Kinesis sequence numbers are numeric decimal strings and are typically 56 characters long, making lexicographic comparison equivalent to numeric ordering for same-width values. However, this is not guaranteed by the Kinesis API contract — LocalStack already returns `Long.MAX_VALUE` (19 digits) as `endSeq` for closed shards.
When `lastSeq` is a 56-digit real sequence number and `endSeq` is 19 digits starting with `9`, the lexicographic comparison `"495..." < "922..."` (`'4' < '9'`) correctly identifies the shard as not fully consumed. That case happens to work, but only by accident of the particular sentinel value.
Consider documenting explicitly that this comparison relies on the assumption that same-shard sequence numbers are always equal-width, or use `BigInteger` comparison for correctness:
```java
// Using BigInteger for guaranteed correct numeric comparison
try {
return new java.math.BigInteger(lastSeq).compareTo(new java.math.BigInteger(endSeq)) >= 0;
} catch (NumberFormatException e) {
return false; // conservative: assume not fully consumed
}
```
— *Greptile* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036407288)) (source:comment#3036407288)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Separator between lastSeq and arrivalTime (epoch millis of the record with last sequence number).
+ * '@' is used as it is absent from numeric Kinesis sequence numbers and visually distinct from '|'.
+ */
+ private static final String ARRIVAL_TIME_SEPARATOR = "@";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq[@arrivalTime][|endSeq],...
+ * For closed shards we store lastSeq|endSeq (or lastSeq@arrivalTime|endSeq with arrival time)
+ * so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map.
+ * Value format: lastSeq, or lastSeq@arrivalTime, or lastSeq|endSeq, or lastSeq@arrivalTime|endSeq.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from a checkpoint value.
+ * Handles formats: "lastSeq", "lastSeq|endSeq", "lastSeq@arrivalTime", "lastSeq@arrivalTime|endSeq".
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int arrivalSep = value.indexOf(ARRIVAL_TIME_SEPARATOR);
+ if (arrivalSep >= 0) {
+ return value.substring(0, arrivalSep);
+ }
+ int endSep = value.indexOf(END_SEQ_SEPARATOR);
+ return endSep >= 0 ? value.substring(0, endSep) : value;
+ }
+
+ /**
+ * Extract arrivalTime (epoch millis) from a checkpoint value if present, otherwise null.
+ * Handles formats: "lastSeq@arrivalTime" and "lastSeq@arrivalTime|endSeq".
+ */
+ public static Long getArrivalTimeFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int arrivalSep = value.indexOf(ARRIVAL_TIME_SEPARATOR);
+ if (arrivalSep < 0) {
+ return null;
+ }
+ String rest = value.substring(arrivalSep + ARRIVAL_TIME_SEPARATOR.length());
+ int endSep = rest.indexOf(END_SEQ_SEPARATOR);
+ String arrivalStr = endSep >= 0 ? rest.substring(0, endSep) : rest;
+ try {
+ return Long.parseLong(arrivalStr);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ /**
+ * Extract endSeq from a checkpoint value if present. Returns null for open shards.
+ * Handles formats: "lastSeq|endSeq" and "lastSeq@arrivalTime|endSeq".
+ * Since '@' and '|' are distinct, '|' always unambiguously marks the endSeq regardless of
+ * whether an arrival time is present.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int endSep = value.indexOf(END_SEQ_SEPARATOR);
+ return endSep >= 0 && endSep < value.length() - 1 ? value.substring(endSep + 1) : null;
+ }
+
+ /**
+ * Parse a checkpoint value into (lastSeq, endSeq). Combines {@link #getLastSeqFromValue} and
+ * {@link #getEndSeqFromValue} into a single call to avoid parsing the value string twice.
+ * @return Pair where left=lastSeq (empty Option when absent), right=endSeq (empty Option for open shards)
+ */
+ public static Pair<Option<String>, Option<String>> parseCheckpointValue(String value) {
+ return Pair.of(Option.ofNullable(getLastSeqFromValue(value)),
+ Option.ofNullable(getEndSeqFromValue(value)));
+ }
+
+ /**
+ * Build checkpoint value without arrival time: "lastSeq" or "lastSeq|endSeq".
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ return buildCheckpointValue(lastSeq, null, endSeq);
+ }
+
+ /**
+ * Build checkpoint value with optional arrival time.
+ * Format: lastSeq[@arrivalTime][|endSeq]
+ */
+ public static String buildCheckpointValue(String lastSeq, Long arrivalTime, String endSeq) {
+ StringBuilder sb = new StringBuilder(lastSeq != null ? lastSeq : "");
+ if (arrivalTime != null) {
+ sb.append(ARRIVAL_TIME_SEPARATOR).append(arrivalTime);
+ }
+ if (endSeq != null && !endSeq.isEmpty()) {
+ sb.append(END_SEQ_SEPARATOR).append(endSeq);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ /**
+ * Returns true when {@code lastCheckpointStr} is a well-formed Kinesis checkpoint for {@code streamName}.
+ * Checks both format (streamName,shardId:seq,...) and that the embedded stream name matches.
+ */
+ public static boolean isValidStreamCheckpoint(Option<String> lastCheckpointStr, String streamName) {
+ return lastCheckpointStr.isPresent()
+ && PATTERN.matcher(lastCheckpointStr.get()).matches()
+ && lastCheckpointStr.get().startsWith(streamName + ",");
+ }
+ }
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // CASE 1: Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+ // CASE 2: Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+ // CASE 3: Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+ // CASE 4: lastSeq < endSeq: may have unread records
+ return true;
+ }
Review Comment:
_⚠️ Potential issue_ | _🟠 Major_
<details>
<summary>🧩 Analysis chain</summary>
🌐 Web query:
`Are AWS Kinesis sequence numbers always the same length within a shard, or can they vary in length?`
💡 Result:
AWS Kinesis sequence numbers can vary in length within a shard. They are strings matching the pattern 0|([1-9]\d{0,128}), where "0" has length 1, and others start with 1-9 followed by 0 to 128 digits, allowing lengths from 1 up to 129 characters.
Citations:
- 1: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html
- 2: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SequenceNumberRange.html
- 3: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html
---
🏁 Script executed:
```shell
find . -type f -name "KinesisOffsetGen.java" | head -5
```
Repository: yihua/hudi
Length of output: 148
---
🏁 Script executed:
```shell
wc -l ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java
```
Repository: yihua/hudi
Length of output: 152
---
🏁 Script executed:
```shell
sed -n '252,270p' ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java
```
Repository: yihua/hudi
Length of output: 745
---
🏁 Script executed:
```shell
sed -n '460,470p' ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java
```
Repository: yihua/hudi
Length of output: 728
---
🏁 Script executed:
```shell
sed -n '494,504p' ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java
```
Repository: yihua/hudi
Length of output: 564
---
🏁 Script executed:
```shell
grep -n "compareTo" ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java | head -20
```
Repository: yihua/hudi
Length of output: 248
---
🏁 Script executed:
```shell
cat -n ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java | sed -n '210,215p'
```
Repository: yihua/hudi
Length of output: 352
---
🏁 Script executed:
```shell
sed -n '215,225p' ./hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java
```
Repository: yihua/hudi
Length of output: 609
---
</details>
**String.compareTo() incorrectly compares Kinesis sequence numbers, affecting unread records and data loss detection.**
Kinesis sequence numbers are 128-bit integers represented as variable-length decimal strings (1–129 characters per AWS API docs). Using `String.compareTo()` performs lexicographic comparison, which produces incorrect results when lengths differ—for example, `"9".compareTo("10") > 0` but numerically `9 < 10`.
This bug occurs at three locations:
- **Line 265** in `hasUnreadRecords()`: Determines if a shard has unread records
- **Line 464**: Detects fully consumed expired shards
- **Line 498**: Detects data loss from trimmed records
Replace all three with numeric comparison using `BigInteger`:
```java
private static int compareSequenceNumbers(String seq1, String seq2) {
return new java.math.BigInteger(seq1).compareTo(new java.math.BigInteger(seq2));
}
```
Then update each location:
- Line 265: `lastSeq.compareTo(endSeq) >= 0` → `compareSequenceNumbers(lastSeq, endSeq) >= 0`
- Line 464: `last.compareTo(endSeqOpt.get()) >= 0` → `compareSequenceNumbers(last, endSeqOpt.get()) >= 0`
- Line 498: `lastSeq.compareTo(shardStartSeq) < 0` → `compareSequenceNumbers(lastSeq, shardStartSeq) < 0`
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java`
around lines 252 - 270, The current logic in KinesisOffsetGen uses
String.compareTo on Kinesis sequence number strings (e.g.,
lastSeq.compareTo(endSeq) and similar checks), which is lexicographic and
incorrect for numeric sequence values; add a helper method
compareSequenceNumbers(String seq1, String seq2) that converts the strings to
java.math.BigInteger and returns their numeric comparison, then replace all
string comparisons: in hasUnreadRecords replace lastSeq.compareTo(endSeq) >= 0
with compareSequenceNumbers(lastSeq, endSeq) >= 0, update the shard-expiration
check that uses last.compareTo(endSeqOpt.get()) >= 0 to use
compareSequenceNumbers(last, endSeqOpt.get()) >= 0, and replace the trimmed-data
detection lastSeq.compareTo(shardStartSeq) < 0 with
compareSequenceNumbers(lastSeq, shardStartSeq) < 0 so all sequence comparisons
are numeric.
```
</details>
<!-- fingerprinting:phantom:poseidon:ocelot:7a2a8930-2f98-4e83-9519-06ffc64e3eaa -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410514)) (source:comment#3036410514)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Separator between lastSeq and arrivalTime (epoch millis of the record with last sequence number).
+ * '@' is used as it is absent from numeric Kinesis sequence numbers and visually distinct from '|'.
+ */
+ private static final String ARRIVAL_TIME_SEPARATOR = "@";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq[@arrivalTime][|endSeq],...
+ * For closed shards we store lastSeq|endSeq (or lastSeq@arrivalTime|endSeq with arrival time)
+ * so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map.
+ * Value format: lastSeq, or lastSeq@arrivalTime, or lastSeq|endSeq, or lastSeq@arrivalTime|endSeq.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from a checkpoint value.
+ * Handles formats: "lastSeq", "lastSeq|endSeq", "lastSeq@arrivalTime", "lastSeq@arrivalTime|endSeq".
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int arrivalSep = value.indexOf(ARRIVAL_TIME_SEPARATOR);
+ if (arrivalSep >= 0) {
+ return value.substring(0, arrivalSep);
+ }
+ int endSep = value.indexOf(END_SEQ_SEPARATOR);
+ return endSep >= 0 ? value.substring(0, endSep) : value;
+ }
+
+ /**
+ * Extract arrivalTime (epoch millis) from a checkpoint value if present, otherwise null.
+ * Handles formats: "lastSeq@arrivalTime" and "lastSeq@arrivalTime|endSeq".
+ */
+ public static Long getArrivalTimeFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int arrivalSep = value.indexOf(ARRIVAL_TIME_SEPARATOR);
+ if (arrivalSep < 0) {
+ return null;
+ }
+ String rest = value.substring(arrivalSep + ARRIVAL_TIME_SEPARATOR.length());
+ int endSep = rest.indexOf(END_SEQ_SEPARATOR);
+ String arrivalStr = endSep >= 0 ? rest.substring(0, endSep) : rest;
+ try {
+ return Long.parseLong(arrivalStr);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ /**
+ * Extract endSeq from a checkpoint value if present. Returns null for open shards.
+ * Handles formats: "lastSeq|endSeq" and "lastSeq@arrivalTime|endSeq".
+ * Since '@' and '|' are distinct, '|' always unambiguously marks the endSeq regardless of
+ * whether an arrival time is present.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int endSep = value.indexOf(END_SEQ_SEPARATOR);
+ return endSep >= 0 && endSep < value.length() - 1 ? value.substring(endSep + 1) : null;
+ }
+
+ /**
+ * Parse a checkpoint value into (lastSeq, endSeq). Combines {@link #getLastSeqFromValue} and
+ * {@link #getEndSeqFromValue} into a single call to avoid parsing the value string twice.
+ * @return Pair where left=lastSeq (empty Option when absent), right=endSeq (empty Option for open shards)
+ */
+ public static Pair<Option<String>, Option<String>> parseCheckpointValue(String value) {
+ return Pair.of(Option.ofNullable(getLastSeqFromValue(value)),
+ Option.ofNullable(getEndSeqFromValue(value)));
+ }
+
+ /**
+ * Build checkpoint value without arrival time: "lastSeq" or "lastSeq|endSeq".
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ return buildCheckpointValue(lastSeq, null, endSeq);
+ }
+
+ /**
+ * Build checkpoint value with optional arrival time.
+ * Format: lastSeq[@arrivalTime][|endSeq]
+ */
+ public static String buildCheckpointValue(String lastSeq, Long arrivalTime, String endSeq) {
+ StringBuilder sb = new StringBuilder(lastSeq != null ? lastSeq : "");
+ if (arrivalTime != null) {
+ sb.append(ARRIVAL_TIME_SEPARATOR).append(arrivalTime);
+ }
+ if (endSeq != null && !endSeq.isEmpty()) {
+ sb.append(END_SEQ_SEPARATOR).append(endSeq);
+ }
+ return sb.toString();
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ /**
+ * Returns true when {@code lastCheckpointStr} is a well-formed Kinesis checkpoint for {@code streamName}.
+ * Checks both format (streamName,shardId:seq,...) and that the embedded stream name matches.
+ */
+ public static boolean isValidStreamCheckpoint(Option<String> lastCheckpointStr, String streamName) {
+ return lastCheckpointStr.isPresent()
+ && PATTERN.matcher(lastCheckpointStr.get()).matches()
+ && lastCheckpointStr.get().startsWith(streamName + ",");
+ }
+ }
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // CASE 1: Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+ // CASE 2: Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+ // CASE 3: Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+ // CASE 4: lastSeq < endSeq: may have unread records
+ return true;
+ }
+ }
+
+ /** Name of the Kinesis Data Stream to consume from (e.g. "my-stream"). */
+ private final String streamName;
+ /** AWS region where the Kinesis stream is provisioned (e.g. "us-east-1"). Required because KinesisClient is region-scoped — each client instance connects to exactly one regional endpoint. */
+ private final String region;
+ /** Optional custom Kinesis endpoint URL, used for localstack or VPC endpoints. Empty when using the default AWS endpoint. */
+ private final Option<String> endpointUrl;
+ /** Strategy that determines where to start reading when no prior checkpoint exists (LATEST, TRIM_HORIZON, etc.). */
+ private final KinesisSourceConfig.KinesisStartingPositionStrategy startingPositionStrategy;
+ /** Raw configuration properties passed from HoodieStreamer; used to resolve Kinesis-specific and shared settings. */
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ this.startingPositionStrategy = KinesisSourceConfig.KinesisStartingPositionStrategy.fromString(
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true));
+ }
+
+ /**
+ * Builds a Kinesis client from explicit parameters. Used by both the instance method
+ * {@link #createKinesisClient()} and by {@link org.apache.hudi.utilities.sources.JsonKinesisSource}
+ * from serializable {@link KinesisReadConfig} in Spark closures.
+ */
+ public static KinesisClient createKinesisClient(String region, String endpointUrl,
+ String accessKey, String secretKey) {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl != null && !endpointUrl.isEmpty()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl));
+ }
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ public KinesisClient createKinesisClient() {
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ return createKinesisClient(region, endpointUrl.orElse(null), accessKey, secretKey);
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long numOpenShards = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, numOpenShards, allShards.size() - numOpenShards);
+ logShardSequenceRanges(allShards);
+ return allShards;
+ }
+
+ /**
+ * Logs each shard's start/end sequence number so they can be used when resetting the checkpoint.
+ */
+ private void logShardSequenceRanges(List<Shard> shards) {
+ for (Shard shard : shards) {
+ String startSeq = (shard.sequenceNumberRange() != null && shard.sequenceNumberRange().startingSequenceNumber() != null)
+ ? shard.sequenceNumberRange().startingSequenceNumber() : "n/a";
+ String endSeq = (shard.sequenceNumberRange() != null && shard.sequenceNumberRange().endingSequenceNumber() != null)
+ ? shard.sequenceNumberRange().endingSequenceNumber() : null;
+ if (endSeq != null) {
+ log.info("Shard {}: startSeq={}, endSeq={} (for checkpoint reset: {}:{}|{})",
+ shard.shardId(), startSeq, endSeq, shard.shardId(), startSeq, endSeq);
+ } else {
+ log.info("Shard {}: startSeq={}, endSeq=open (for checkpoint reset from start: {}:{})",
+ shard.shardId(), startSeq, shard.shardId(), startSeq);
+ }
+ }
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ long numEvents = calculateNumEvents(sourceLimit, props);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // STEP 1: List all open and closed shards from the server.
+ // Note: no expired shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+ // STEP 2: parse last checkpoint if exists.
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+ if (CheckpointUtils.isValidStreamCheckpoint(lastCheckpointStr, streamName)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty()) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ // This is important to detect data loss.
+ if (!expiredShardIds.isEmpty()) {
+ checkDataLossOnExpiredShards(expiredShardIds, checkpointOffsets);
+ }
+ // Handle regular case.
+ // Parse lastSeq for open and closed shards.
+ // For closed shards, even if all their records have been consumed, they are still included.
+ for (String shardId : availableShardIds) {
+ if (checkpointOffsets.containsKey(shardId)) {
+ Option<String> lastSeqOpt = Option.ofNullable(
+ CheckpointUtils.getLastSeqFromValue(checkpointOffsets.get(shardId)))
+ .flatMap(seq -> seq.isEmpty() ? Option.empty() : Option.of(seq));
+ lastSeqOpt.ifPresent(seq -> fromSequenceNumbers.put(shardId, seq));
+ }
+ }
+ // Check if any available shard's checkpoint lastSeq has fallen behind the shard's
+ // trim horizon, i.e., records between lastSeq and the earliest available seq were trimmed.
+ checkDataLossOnAvailableShards(shards, fromSequenceNumbers);
+ }
+ }
+ // STEP 3: Create ranges.
+ List<KinesisShardRange> ranges = new ArrayList<>();
+ for (Shard shard : shards) {
+ String shardId = shard.shardId();
+ Option<String> startSeq = fromSequenceNumbers.containsKey(shardId)
+ ? Option.of(fromSequenceNumbers.get(shardId))
+ : Option.empty();
+ Option<String> endSeq = shard.sequenceNumberRange() != null
+ ? Option.ofNullable(shard.sequenceNumberRange().endingSequenceNumber())
+ : Option.empty();
+ ranges.add(KinesisShardRange.of(shardId, startSeq, endSeq));
+ }
+ log.info("About to read up to {} events from {} shards in stream {})",
+ numEvents, ranges.size(), streamName);
Review Comment:
_⚠️ Potential issue_ | _🟡 Minor_
**Typo in log message: unmatched closing parenthesis.**
The log message has an extra `)` at the end that appears unintentional.
<details>
<summary>✏️ Proposed fix</summary>
```diff
- log.info("About to read up to {} events from {} shards in stream {})",
+ log.info("About to read up to {} events from {} shards in stream {}",
numEvents, ranges.size(), streamName);
```
</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
log.info("About to read up to {} events from {} shards in stream {}",
numEvents, ranges.size(), streamName);
```
</details>
<!-- suggestion_end -->
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java`
around lines 433 - 434, The log message in KinesisOffsetGen's log.info call has
an extra closing parenthesis at the end of the format string; update the
log.info invocation (the line that logs "About to read up to {} events from {}
shards in stream {})") to remove the stray ')' so the format string is "About to
read up to {} events from {} shards in stream {}", keeping the same arguments
(numEvents, ranges.size(), streamName).
```
</details>
<!-- fingerprinting:phantom:poseidon:ocelot:7a2a8930-2f98-4e83-9519-06ffc64e3eaa -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410517)) (source:comment#3036410517)
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -3112,6 +3120,348 @@ public void testJsonKafkaDFSSource() throws Exception {
deltaStreamer.shutdownGracefully();
}
+ /**
+ * Tests DeltaStreamer ingestion from JsonKinesisSource using LocalStack.
+ */
+ @Nested
+ @org.junit.jupiter.api.TestInstance(org.junit.jupiter.api.TestInstance.Lifecycle.PER_CLASS)
+ class TestKinesisSource {
+
+ private KinesisTestUtils kinesisTestUtils;
+
+ @BeforeAll
+ void initKinesis() {
+ kinesisTestUtils = new KinesisTestUtils().setup();
+ }
+
+ @AfterAll
+ void tearDownKinesis() {
+ if (kinesisTestUtils != null) {
+ kinesisTestUtils.teardown();
+ }
+ }
+
+ @Test
+ public void testJsonKinesisDFSSource() throws Exception {
+ String streamName = "test-kinesis-stream-" + testNum;
+ // Create stream with at least 2 shards to exercise multi-shard reading
+ prepareJsonKinesisDFSFiles(JSON_KINESIS_NUM_RECORDS, true, streamName);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, LocalStackJsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+
+ // Track commit completion times for incremental queries
+ List<String> commitCompletionTimes = new ArrayList<>();
+
+ // Batch 1: initial ingestion
+ deltaStreamer.sync();
+ assertRecordCount(JSON_KINESIS_NUM_RECORDS, tableBasePath, sqlContext);
+ String commit1 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit1, JSON_KINESIS_NUM_RECORDS);
+
+ int totalRecords = JSON_KINESIS_NUM_RECORDS;
+ // Batch 2
+ int batch2Records = 10;
+ totalRecords += batch2Records;
+ prepareJsonKinesisDFSFiles(batch2Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit2 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit2, batch2Records);
+
+ // Batch 3
+ int batch3Records = 8;
+ totalRecords += batch3Records;
+ prepareJsonKinesisDFSFiles(batch3Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit3 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit3, batch3Records);
+
+ // Batch 4
+ int batch4Records = 7;
+ totalRecords += batch4Records;
+ prepareJsonKinesisDFSFiles(batch4Records, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(totalRecords, tableBasePath, sqlContext);
+ String commit4 = getLatestCommitInstantTime(tableBasePath);
+ commitCompletionTimes.add(getCommitCompletionTime(tableBasePath));
+ assertRecordsForCommit(tableBasePath, commit4, batch4Records);
+ deltaStreamer.shutdownGracefully();
+
+ // Incremental queries for a subset of commits
+ sqlContext.clearCache();
+ String c1 = commitCompletionTimes.get(0);
+ String c2 = commitCompletionTimes.get(1);
+ String c3 = commitCompletionTimes.get(2);
+ String c4 = commitCompletionTimes.get(3);
+
+ // Incremental: first commit only (batch 1)
+ Dataset<Row> incrBatch1 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), "000")
+ .option(DataSourceReadOptions.END_COMMIT().key(), c1)
+ .load(tableBasePath);
+ assertEquals(JSON_KINESIS_NUM_RECORDS, incrBatch1.count(), "Incremental read for batch 1");
+
+ // Incremental: commits 2 and 3 (batches 2 + 3)
+ Dataset<Row> incrBatches2And3 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), c1)
+ .option(DataSourceReadOptions.END_COMMIT().key(), c3)
+ .load(tableBasePath);
+ assertEquals(batch2Records + batch3Records, incrBatches2And3.count(), "Incremental read for batches 2 and 3");
+
+ // Incremental: commits 3 and 4 (batches 3 + 4)
+ Dataset<Row> incrBatches3And4 = sqlContext.read().format("org.apache.hudi")
+ .options(hudiOpts)
+ .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
+ .option(DataSourceReadOptions.START_COMMIT().key(), c2)
+ .option(DataSourceReadOptions.END_COMMIT().key(), c4)
+ .load(tableBasePath);
+ assertEquals(batch3Records + batch4Records, incrBatches3And4.count(), "Incremental read for batches 3 and 4");
+
+ // Verify data correctness: read back and check _row_key exists
+ sqlContext.clearCache();
+ Dataset<Row> ds = sqlContext.read().format("org.apache.hudi").load(tableBasePath);
+ assertEquals(totalRecords, ds.count());
+ assertTrue(ds.filter("_row_key is not null").count() > 0);
+ }
+
+ @Test
+ public void testJsonKinesisAggregatedRecords() throws Exception {
+ String streamName = "test-kinesis-stream-agg-" + testNum;
+ kinesisTestUtils.createStream(streamName, 2);
+ HoodieTestDataGenerator dataGenerator = new HoodieTestDataGenerator(System.nanoTime());
+ String[] jsonRecords = UtilitiesTestBase.Helpers.jsonifyRecords(
+ dataGenerator.generateInsertsAsPerSchema("000", 6, HoodieTestDataGenerator.TRIP_SCHEMA));
+ kinesisTestUtils.sendAggregatedRecords(streamName, jsonRecords);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_agg_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, LocalStackJsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+ deltaStreamer.sync();
+ assertRecordCount(6, tableBasePath, sqlContext);
+ deltaStreamer.shutdownGracefully();
+ testNum++;
+ }
+
+ @Test
+ public void testJsonKinesisShardSplitCheckpoint() throws Exception {
+ String streamName = "test-kinesis-stream-split-" + testNum;
+ kinesisTestUtils.createStream(streamName, 2);
+ prepareJsonKinesisDFSFiles(10, false, streamName);
+ prepareJsonKinesisDFSSource(PROPS_FILENAME_TEST_JSON_KINESIS, streamName);
+ String tableBasePath = basePath + "/test_json_kinesis_split_table" + testNum;
+ HoodieDeltaStreamer deltaStreamer = new HoodieDeltaStreamer(
+ TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT, LocalStackJsonKinesisSource.class.getName(),
+ Collections.emptyList(), PROPS_FILENAME_TEST_JSON_KINESIS, false,
+ true, 100000, false, null, null, "timestamp", null), jsc);
+ deltaStreamer.sync();
+ assertRecordCount(10, tableBasePath, sqlContext);
+ String checkpointAfterBatch1 = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterBatch1);
+ assertTrue(checkpointAfterBatch1.startsWith(streamName + ","));
+ assertTrue(checkpointAfterBatch1.contains(":"), "Checkpoint should have shard:seq format");
+
+ kinesisTestUtils.updateShardCount(streamName, 4);
+ prepareJsonKinesisDFSFiles(5, false, streamName);
+ deltaStreamer.sync();
+ assertRecordCount(15, tableBasePath, sqlContext);
+ String checkpointAfterSplit = getCheckpointFromLatestCommit(tableBasePath);
+ assertNotNull(checkpointAfterSplit);
+ assertTrue(checkpointAfterSplit.startsWith(streamName + ","));
+ // Closed parent shards must have lastSeq|endSeq with endSeq <= lastSeq so we can detect
+ // "fully consumed" when parent expires. LocalStack returns Long.MAX_VALUE; we replace with lastSeq.
+ assertFalse(checkpointAfterSplit.contains("9223372036854775807"),
+ "Checkpoint should not contain Long.MAX_VALUE as endSeq (parent expiry would fail)");
+ // Verify closed shard checkpoint format: lastSeq|endSeq with lastSeq >= endSeq (fully consumed)
+ Map<String, String> offsets = KinesisOffsetGen.CheckpointUtils.strToOffsets(checkpointAfterSplit);
+ for (Map.Entry<String, String> e : offsets.entrySet()) {
+ Pair<Option<String>, Option<String>> seqs = KinesisOffsetGen.CheckpointUtils.parseCheckpointValue(e.getValue());
+ String lastSeq = seqs.getLeft().orElse(null);
+ String endSeq = seqs.getRight().orElse(null);
+ if (endSeq != null && lastSeq != null && !lastSeq.isEmpty()) {
+ assertTrue(lastSeq.compareTo(endSeq) >= 0,
+ "Closed shard " + e.getKey() + ": lastSeq should be >= endSeq (fully consumed)");
Review Comment:
_⚠️ Potential issue_ | _🟡 Minor_
**Compare sequence numbers numerically, not lexicographically.**
Lines 3295 and 3383 use `String.compareTo`, which is lexicographic. If Kinesis emits sequence numbers with different widths, the assertion can treat `"9"` as greater than `"10"` and misclassify a closed shard as fully consumed.
Also applies to: 3377-3384
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java`
around lines 3289 - 3296, The assertion currently compares Kinesis sequence
numbers as strings using String.compareTo which is lexicographic and incorrect
for numeric sequence IDs; change the comparison to numeric by parsing the
sequence values (from KinesisOffsetGen.CheckpointUtils.strToOffsets /
parseCheckpointValue output: lastSeq and endSeq) into a numeric type (use
BigInteger to be safe for very large sequence IDs, or Long if you confirm
bounds) and then compare numerically (e.g., bigLast.compareTo(bigEnd) >= 0).
Apply the same numeric-parse-and-compare fix to both occurrences that use
String.compareTo to ensure closed shards are validated correctly.
```
</details>
<!-- fingerprinting:phantom:medusa:grasshopper:2d6277d7-75f4-4e28-b60a-1efa18b71895 -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410529)) (source:comment#3036410529)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.calculateNumEvents;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ // UTC in milliseconds.
+ private final Option<Long> lastArrivalTime;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /**
+ * Per-shard fetch result stored in the persisted RDD.
+ * Records are eagerly materialized as List<String> so that both fields survive RDD spill
+ * to disk: List and ShardFetchSummary are fully serializable, unlike a transient Iterable.
+ */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final ShardFetchSummary summary;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+ /** Arrival time (epoch millis) of the record with last sequence number, per shard. */
+ protected Map<String, Long> lastArrivalTimes;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges, long sourceLimit) {
+ long numEvents = calculateNumEvents(sourceLimit, props);
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPositionStrategy(),
+ shouldAddMetaFields,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_MAX_RECORDS_PER_REQUEST),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // NOTE that: Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, numEvents / shardRanges.length) : numEvents,
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_RETRY_INITIAL_INTERVAL_MS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_RETRY_MAX_INTERVAL_MS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_THROTTLE_TIMEOUT_MS));
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = KinesisOffsetGen.createKinesisClient(
+ readConfig.getRegion(), readConfig.getEndpointUrl(),
+ readConfig.getAccessKey(), readConfig.getSecretKey())) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ // Lazy iterator: fetches one GetRecords page at a time, keeping only one page in
+ // executor memory instead of the full shard batch. Records are GC-eligible as soon
+ // as they are converted to JSON strings below.
+ KinesisSource.ShardRecordIterator recordIt = KinesisSource.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMilliSeconds(),
+ readConfig.getMaxRecordsPerShard(), readConfig.isEnableDeaggregation(),
+ readConfig.getRetryInitialIntervalMs(), readConfig.getRetryMaxIntervalMs(),
+ readConfig.getThrottleTimeoutMs());
+
+ String shardId = range.getShardId();
+ boolean addMetaFields = readConfig.isShouldAddMetaFields();
+ List<String> jsonRecords = new ArrayList<>();
+ long numNull = 0;
+ java.time.Instant lastArrivalTimestamp = null;
+ while (recordIt.hasNext()) {
+ Record r = recordIt.next();
+ lastArrivalTimestamp = r.approximateArrivalTimestamp();
+ String s = recordToJsonStatic(r, shardId, addMetaFields);
+ if (s != null) {
+ jsonRecords.add(s);
+ } else {
+ numNull++;
+ }
+ }
+ if (numNull > 0) {
+ log.warn("There are {} null strings for shard id {}", numNull, shardId);
+ }
+ // Capture the arrival time of the last record (same record whose sequence number
+ // becomes the checkpoint lastSeq) so it can be embedded in the checkpoint.
+ Option<Long> lastArrivalTime = lastArrivalTimestamp != null
+ ? Option.of(lastArrivalTimestamp.toEpochMilli()) : Option.empty();
+ // recordCount reflects actual output records (null-filtered), not raw Kinesis count.
+ // NOTE: getLastSequenceNumber/isReachedEndOfShard are final only after hasNext()==false.
+ ShardFetchSummary summary = new ShardFetchSummary(shardId,
+ recordIt.getLastSequenceNumber(), lastArrivalTime,
+ jsonRecords.size(), recordIt.isReachedEndOfShard());
+ results.add(new ShardFetchResult(jsonRecords, summary));
+ }
+ }
+ return results.iterator();
+ });
+
+ if (persistedFetchRdd != null) {
+ persistedFetchRdd.unpersist();
+ persistedFetchRdd = null;
+ }
+ boolean persistFetchRdd = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_PERSIST_FETCH_RDD);
+ if (persistFetchRdd) {
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+ } else {
+ log.debug("{} is false: fetch RDD is not persisted. The same Kinesis fetch may run twice (checkpoint + "
+ + "record write), which can cause duplicate records to be written. Set to true for correct behavior.",
+ KinesisSourceConfig.KINESIS_PERSIST_FETCH_RDD.key());
+ }
+ // Guard: if anything below throws, unpersist immediately so the cached RDD doesn't leak.
+ boolean succeeded = false;
+ try {
+ // Collect basic information that will be used to construct the final checkpoint.
+ collectCheckpointInfo(fetchRdd);
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long manualPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MANUAL_PARTITIONS);
+ if (manualPartitions > 0) {
+ int targetPartitions = (int) manualPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (manualPartitions={})",
+ shardRanges.length, targetPartitions, manualPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+ succeeded = true;
+ return recordRdd;
+ } finally {
+ if (!succeeded) {
+ releaseResources();
+ }
+ }
+ }
+
+ private static String recordToJsonStatic(Record record, String shardId, boolean shouldAddMetaFields) {
+ String dataStr = record.data().asUtf8String();
+ // Pure empty or null records in Kinesis is not meaningful.
+ if (dataStr == null || dataStr.trim().isEmpty()) {
+ return null;
+ }
+ if (shouldAddMetaFields) {
+ try {
+ ObjectNode node = (ObjectNode) OBJECT_MAPPER.readTree(dataStr);
+ node.put("_hoodie_kinesis_source_sequence_number", record.sequenceNumber());
+ node.put("_hoodie_kinesis_source_shard_id", shardId);
+ node.put("_hoodie_kinesis_source_partition_key", record.partitionKey());
+ if (record.approximateArrivalTimestamp() != null) {
+ node.put("_hoodie_kinesis_source_timestamp",
+ record.approximateArrivalTimestamp().toEpochMilli());
+ }
+ return OBJECT_MAPPER.writeValueAsString(node);
+ } catch (Exception e) {
+ // We can disable the flag for mitigation.
+ throw new HoodieException("Failed to add metadata fields", e);
+ }
+ }
+ return dataStr;
+ }
Review Comment:
_⚠️ Potential issue_ | _🟠 Major_
<details>
<summary>🧩 Analysis chain</summary>
🏁 Script executed:
```shell
#!/bin/bash
# Check if there's documentation or other handling for JSON array payloads in Kinesis sources
rg -n -C3 "ObjectNode|JsonNode" --type=java hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/
```
Repository: yihua/hudi
Length of output: 10606
---
🏁 Script executed:
```shell
#!/bin/bash
# Search for JsonKinesisSource tests
find . -name "*JsonKinesisSource*Test*.java" -o -name "*Test*JsonKinesisSource*.java"
```
Repository: yihua/hudi
Length of output: 145
---
🏁 Script executed:
```shell
#!/bin/bash
# Check if there are any type checks or validations in JsonKinesisSource
rg -n "isObject|isArray|getNodeType" hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java
```
Repository: yihua/hudi
Length of output: 36
---
🏁 Script executed:
```shell
#!/bin/bash
# Look for documentation or comments about JSON payload types
rg -n -B2 -A2 "JSON|array|object|payload" hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java | head -50
```
Repository: yihua/hudi
Length of output: 1084
---
🏁 Script executed:
```shell
#!/bin/bash
# Check JsonKafkaSource to see if it has the same limitation or handles it differently
sed -n '100,130p' hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java
```
Repository: yihua/hudi
Length of output: 1629
---
🏁 Script executed:
```shell
#!/bin/bash
# Examine the test file to see if arrays are tested
cat hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJsonKinesisSource.java
```
Repository: yihua/hudi
Length of output: 21272
---
🏁 Script executed:
```shell
#!/bin/bash
# Get the full catch block context from JsonKinesisSource
sed -n '226,243p' hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java
```
Repository: yihua/hudi
Length of output: 834
---
🏁 Script executed:
```shell
#!/bin/bash
# Get the full catch block context from JsonKafkaSource for comparison (around line 116)
sed -n '110,135p' hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java
```
Repository: yihua/hudi
Length of output: 1172
---
</details>
**JSON array payloads will throw when metadata fields are enabled.**
When `shouldAddMetaFields` is `true`, the code attempts an unchecked cast to `ObjectNode` (line 228). If the Kinesis record contains valid JSON that is not an object—such as a JSON array `[1,2,3]`—this will throw a `ClassCastException` wrapped in a `HoodieException`.
Compare this to `JsonKafkaSource`, which handles the same pattern differently: it catches exceptions during this operation and gracefully returns the original record. The test file `TestJsonKinesisSource.java` also suggests this behavior may be unintended, with test `testRecordToJsonInvalidJsonWithShouldAddOffsetsReturnsOriginalString()` expecting the original string to be returned rather than an exception thrown.
Consider checking the node type before casting (e.g., `node.isObject()`) to align with either explicit documentation of supported types or with the error-handling approach used in `JsonKafkaSource`.
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java`
around lines 226 - 243, The code inside the shouldAddMetaFields block blindly
casts OBJECT_MAPPER.readTree(dataStr) to ObjectNode which throws for non-object
JSON (e.g., arrays); change the logic in JsonKinesisSource so after reading the
JsonNode you check node.isObject() before casting and only add the meta fields
(using node.put(...) and OBJECT_MAPPER.writeValueAsString(node)) when it is an
object; if it is not an object (or if readTree fails), mirror JsonKafkaSource
behavior by catching exceptions or returning the original dataStr instead of
throwing a HoodieException, keeping references to shouldAddMetaFields,
OBJECT_MAPPER.readTree, ObjectNode, and the record.* methods (sequenceNumber,
partitionKey, approximateArrivalTimestamp) for locating the code to modify.
```
</details>
<!-- fingerprinting:phantom:poseidon:ocelot:905b5bed-b2bc-4566-aabc-7fe1d943d3ff -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410523)) (source:comment#3036410523)
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisDeaggregator;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+
+@Slf4j
+public abstract class KinesisSource<T> extends Source<T> {
+
+ protected static final String METRIC_NAME_KINESIS_MESSAGE_IN_COUNT = "kinesisMessageInCount";
+
+ protected final HoodieIngestionMetrics metrics;
+ protected final SchemaProvider schemaProvider;
+ protected KinesisOffsetGen offsetGen;
+ protected final boolean shouldAddMetaFields;
+ /** Checkpoint data (shardId -> sequenceNumber) collected during toBatch execution. Set by subclasses. */
+ protected Map<String, String> lastCheckpointData;
+
+ protected KinesisSource(TypedProperties props, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SourceType sourceType, HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(props, sparkContext, sparkSession, sourceType, streamContext);
+ this.schemaProvider = streamContext.getSchemaProvider();
+ this.metrics = metrics;
+ this.shouldAddMetaFields = getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_APPEND_OFFSETS);
+ }
+
+ @Override
+ protected final InputBatch<T> fetchNewData(Option<String> lastCkptStr, long sourceLimit) {
+ throw new UnsupportedOperationException("KinesisSource#fetchNewData should not be called");
+ }
+
+ @Override
+ protected InputBatch<T> readFromCheckpoint(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
+ // STEP 1: Collect all available shards for the stream: open/closed shards.
+ KinesisOffsetGen.KinesisShardRange[] allOpenClosedShardRanges = offsetGen.getNextShardRanges(lastCheckpoint, sourceLimit);
+ // STEP 2: Filter out shards with no unread records to avoid unnecessary GetRecords calls.
+ boolean useLatestStartingPositionStrategy =
+ offsetGen.getStartingPositionStrategy() == KinesisSourceConfig.KinesisStartingPositionStrategy.LATEST;
+ int numShardsBeforeFilter = allOpenClosedShardRanges.length;
+ KinesisOffsetGen.KinesisShardRange[] shardRangesWithUnreadRecords = Arrays.stream(allOpenClosedShardRanges)
+ .filter(range -> range.hasUnreadRecords(useLatestStartingPositionStrategy))
+ .toArray(KinesisOffsetGen.KinesisShardRange[]::new);
+ if (numShardsBeforeFilter > shardRangesWithUnreadRecords.length) {
+ log.info("Filtered {} shards with no unread records, {} shards remain",
+ numShardsBeforeFilter - shardRangesWithUnreadRecords.length, shardRangesWithUnreadRecords.length);
+ }
+ // When nothing to read, return empty batch and previous checkpoint if any.
+ if (shardRangesWithUnreadRecords.length == 0) {
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, 0);
+ String checkpointStr = lastCheckpoint.isPresent() ? lastCheckpoint.get().getCheckpointKey() : "";
+ return new InputBatch<>(Option.empty(), checkpointStr);
+ }
+ // STEP 3: Otherwise, do the read.
+ T batch = toBatch(shardRangesWithUnreadRecords, sourceLimit);
+ // STEP 4: Generate checkpoint.
+ // Pass allOpenClosedShardRanges so filtered-out shards are preserved in the checkpoint; otherwise
+ // next run would re-read them from TRIM_HORIZON and cause duplicates
+ String checkpointStr = createCheckpointFromBatch(batch, shardRangesWithUnreadRecords, allOpenClosedShardRanges);
+ // STEP 5: Emit metrics.
+ long totalMsgs = getRecordCount(batch);
+ metrics.updateStreamerSourceNewMessageCount(METRIC_NAME_KINESIS_MESSAGE_IN_COUNT, totalMsgs);
+ log.info("Read {} records from Kinesis stream {} with {} shards, checkpoint: {}",
+ totalMsgs, offsetGen.getStreamName(), shardRangesWithUnreadRecords.length, checkpointStr);
+
+ return new InputBatch<>(Option.of(batch), checkpointStr);
+ }
+
+ /** Upper bound on consecutive empty GetRecords responses before giving up on a shard. */
+ private static final int MAX_EMPTY_RESPONSES_FROM_GET_RECORDS = 100;
+
+ /**
+ * Lazy iterator over records from a single Kinesis shard.
+ *
+ * <p>Records are fetched one GetRecords page at a time; the next page is only requested once all
+ * records from the current page have been consumed. This avoids holding the full shard batch in
+ * executor memory simultaneously with the caller's output collection.
+ *
+ * <p>After {@link #hasNext()} returns {@code false} callers must read
+ * {@link #getLastSequenceNumber()} and {@link #isReachedEndOfShard()} to obtain checkpoint state.
+ *
+ * <p><b>lastSequenceNumber correctness invariant:</b> the sequence number is taken from the last
+ * <em>raw</em> Kinesis record (pre-deaggregation) of a page and is only committed once all
+ * deaggregated records from that page have been yielded. This guarantees the checkpoint never
+ * advances past records that have not yet been returned to the caller.
+ */
+ public static class ShardRecordIterator implements Iterator<Record> {
+ private final KinesisClient client;
+ private final String shardId;
+ private final int maxRecordsPerRequest;
+ private final long requestIntervalMs;
+ private final long maxTotalRecords;
+ private final boolean enableDeaggregation;
+ private final long retryInitialIntervalMs;
+ private final long retryMaxIntervalMs;
+ private final long throttleTimeoutMs;
+
+ /** Current position in the Kinesis shard; null means the shard is exhausted. */
+ private String shardIteratorStr;
+ /** Records from the most recently fetched page, ready to be yielded. */
+ private Iterator<Record> currentPage = Collections.emptyIterator();
+ /**
+ * Raw lastSeq of the page currently being consumed. Moved to {@link #lastSequenceNumber} only
+ * when the page iterator is fully exhausted, ensuring the checkpoint never skips records.
+ */
+ private String pendingPageLastSeq = null;
+ /** Checkpoint-safe lastSeq: reflects only fully-consumed pages. */
+ private String lastSequenceNumber = null;
+ private boolean reachedEndOfShard = false;
+ /** True once no further GetRecords calls should be made. */
+ private boolean fetchingDone = false;
+ private long totalConsumed = 0;
+ private int emptyPageCount = 0;
+
+ /**
+ * Dynamically tuned records-per-request limit.
+ * Halved on each ProvisionedThroughputExceededException and held there for the rest of the shard read.
+ */
+ private int currentMaxRecords;
+ /** Epoch ms of the last successful GetRecords call; used to enforce {@link #throttleTimeoutMs}. */
+ private long lastSuccessTimeMs;
+
+ public ShardRecordIterator(String initialShardIterator, KinesisClient client, String shardId,
+ int maxRecordsPerRequest, long requestIntervalMs, long maxTotalRecords, boolean enableDeaggregation,
+ long retryInitialIntervalMs, long retryMaxIntervalMs, long throttleTimeoutMs) {
+ this.shardIteratorStr = initialShardIterator;
+ this.client = client;
+ this.shardId = shardId;
+ this.maxRecordsPerRequest = maxRecordsPerRequest;
+ this.requestIntervalMs = requestIntervalMs;
+ this.maxTotalRecords = maxTotalRecords;
+ this.enableDeaggregation = enableDeaggregation;
+ this.retryInitialIntervalMs = retryInitialIntervalMs;
+ this.retryMaxIntervalMs = retryMaxIntervalMs;
+ this.throttleTimeoutMs = throttleTimeoutMs;
+ this.currentMaxRecords = maxRecordsPerRequest;
+ this.lastSuccessTimeMs = System.currentTimeMillis();
+ }
+
+ @Override
+ public boolean hasNext() {
+ while (true) {
+ if (currentPage.hasNext()) {
+ return true;
+ }
+ // Current page fully consumed: commit its lastSeq before moving on.
+ commitPendingPageLastSeq();
+ if (fetchingDone) {
+ return false;
+ }
+ fetchNextPage();
+ // Loop: if the page was empty, try fetching again (up to MAX_EMPTY_RESPONSES limit).
+ }
+ }
+
+ @Override
+ public Record next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException("No more records for shard " + shardId);
+ }
+ totalConsumed++;
+ return currentPage.next();
+ }
+
+ public Option<String> getLastSequenceNumber() {
+ return Option.ofNullable(lastSequenceNumber);
+ }
+
+ public boolean isReachedEndOfShard() {
+ return reachedEndOfShard;
+ }
+
+ private void commitPendingPageLastSeq() {
+ if (pendingPageLastSeq != null) {
+ lastSequenceNumber = pendingPageLastSeq;
+ pendingPageLastSeq = null;
+ }
+ }
+
+ private void fetchNextPage() {
+ if (shardIteratorStr == null || totalConsumed >= maxTotalRecords) {
+ fetchingDone = true;
+ return;
+ }
+ GetRecordsResponse response;
+ int attempt = 0;
+ while (true) {
+ try {
+ response = client.getRecords(
+ GetRecordsRequest.builder()
+ .shardIterator(shardIteratorStr)
+ .limit(Math.min(currentMaxRecords, (int) (maxTotalRecords - totalConsumed)))
+ .build());
Review Comment:
_⚠️ Potential issue_ | _🟠 Major_
**Compute the per-request limit in `long` before narrowing it.**
Line 240 casts `maxTotalRecords - totalConsumed` to `int` before taking the min. If the remaining quota is larger than `Integer.MAX_VALUE`, that wraps negative and sends an invalid `limit` to `GetRecords`, so large-but-valid source limits fail at runtime.
<details>
<summary>Suggested fix</summary>
```diff
response = client.getRecords(
GetRecordsRequest.builder()
.shardIterator(shardIteratorStr)
- .limit(Math.min(currentMaxRecords, (int) (maxTotalRecords - totalConsumed)))
+ .limit((int) Math.min((long) currentMaxRecords, maxTotalRecords - totalConsumed))
.build());
```
</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
response = client.getRecords(
GetRecordsRequest.builder()
.shardIterator(shardIteratorStr)
.limit((int) Math.min((long) currentMaxRecords, maxTotalRecords - totalConsumed))
.build());
```
</details>
<!-- suggestion_end -->
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/KinesisSource.java`
around lines 237 - 241, In KinesisSource, compute the remaining per-request
limit as a long before narrowing to int to avoid overflow: calculate remaining =
maxTotalRecords - totalConsumed (as long), then compute perRequestLimitLong =
Math.min(currentMaxRecords, remaining) and finally cast perRequestLimitLong to
int only when building GetRecordsRequest.limit; update the
GetRecordsRequest.builder() call (the code using shardIteratorStr,
currentMaxRecords, maxTotalRecords, totalConsumed) to use this int-cast of the
safe long-min result so you never pass a negative/overflowed int to
GetRecordsRequest.
```
</details>
<!-- fingerprinting:phantom:medusa:grasshopper:2d6277d7-75f4-4e28-b60a-1efa18b71895 -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/19#discussion_r3036410526)) (source:comment#3036410526)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034687550
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
Review Comment:
yes. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185930154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064)
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4015477564
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* a28e9e73447bf984536783a76949fd1f98f531dc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227)
* 759d3cdbf5684af44f12fb2ec7ef40c38760ab23 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913793905
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
+ // Check for expired shards (checkpoint references shards no longer in stream, e.g., past retention)
+ Set<String> availableShardIds = shards.stream().map(Shard::shardId).collect(Collectors.toSet());
+ List<String> expiredShardIds = checkpointOffsets.keySet().stream()
+ .filter(id -> !availableShardIds.contains(id))
+ .collect(Collectors.toList());
+ // Handle expired shards that exist in the last checkpoint.
+ if (!expiredShardIds.isEmpty()) {
+ boolean failOnDataLoss = getBooleanWithAltKeys(props, KinesisSourceConfig.ENABLE_FAIL_ON_DATA_LOSS);
+ for (String shardId : expiredShardIds) {
+ String value = checkpointOffsets.get(shardId);
+ String lastSeq = CheckpointUtils.getLastSeqFromValue(value);
+ String endSeq = CheckpointUtils.getEndSeqFromValue(value);
+ boolean fullyConsumed;
+ if (endSeq != null) {
+ // CASE 1: lastSeq >= endSeq: all records have been consumed.
+ fullyConsumed = lastSeq != null && lastSeq.compareTo(endSeq) >= 0;
+ } else {
+ // CASE 2: lastSeq < endSeq: some records haven't been consumed.
+ // CASE 3: endSeq == null: open shard.
+ fullyConsumed = false;
+ }
Review Comment:
Slightly different here is that we don't need to worry about start point strategy. Meanwhile, make it more concise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4028392596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513)
* 8d77ec07f756f71fb3105c8a38008a995b05d793 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2843313206
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
Review Comment:
```suggestion
public static boolean isStreamCheckpointValid(Option<String> lastCheckpointStr) {
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
Review Comment:
Add `lastCheckpointStr.get().startsWith(streamName + ",")` to `CheckpointUtils#checkStreamCheckpoint`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-3944739338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182)
* 917c414dba55734c588d3c43c4b0e16e137fb211 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185427682
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 5aaba50d2b78079d344e8d39c6baf0e76da9f5ae Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557)
* 0980ea929e91ae9637a27f24f07f00bf2ed8db4d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4186021094
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034649407
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
.markAdvanced()
.withDocumentation("When enabled, appends Kinesis metadata (sequence number, shard id, arrival timestamp, partition key) to records.");
public static final ConfigProperty<Boolean> KINESIS_ENABLE_DEAGGREGATION = ConfigProperty
.key(PREFIX + "enable.deaggregation")
.defaultValue(true)
- .withAlternatives(OLD_PREFIX + "enable.deaggregation")
.markAdvanced()
.withDocumentation("When enabled, de-aggregates records produced by Kinesis Producer Library (KPL). "
+ "Non-aggregated records pass through unchanged. Set to false if producers do not use KPL.");
public static final ConfigProperty<Boolean> ENABLE_FAIL_ON_DATA_LOSS = ConfigProperty
- .key(PREFIX + "enable.failOnDataLoss")
+ .key(PREFIX + "enable.fail.on.data.loss")
.defaultValue(false)
- .withAlternatives(OLD_PREFIX + "enable.failOnDataLoss")
.markAdvanced()
- .withDocumentation("Fail when checkpoint references an expired shard instead of seeking to TRIM_HORIZON.");
+ .withDocumentation("Fail when checkpoint references an expired shard which has not been fully consumed.");
public static final ConfigProperty<String> KINESIS_STARTING_POSITION = ConfigProperty
.key(PREFIX + "starting.position")
.defaultValue("LATEST")
- .withAlternatives(OLD_PREFIX + "starting.position")
.markAdvanced()
- .withDocumentation("Starting position when no checkpoint exists. TRIM_HORIZON (or EARLIEST), or LATEST. Default: LATEST.");
+ .withDocumentation("Starting position when no checkpoint exists. EARLIEST or LATEST. Default: LATEST.");
- public static final ConfigProperty<Integer> KINESIS_GET_RECORDS_MAX_RECORDS = ConfigProperty
- .key(PREFIX + "getRecords.maxRecords")
+ public static final ConfigProperty<Integer> KINESIS_MAX_RECORDS_PER_REQUEST = ConfigProperty
+ .key(PREFIX + "max.records.per.request")
.defaultValue(10000)
- .withAlternatives(OLD_PREFIX + "getRecords.maxRecords")
.markAdvanced()
.withDocumentation("Maximum number of records to fetch per GetRecords API call. Kinesis limit is 10000.");
public static final ConfigProperty<Long> KINESIS_GET_RECORDS_INTERVAL_MS = ConfigProperty
- .key(PREFIX + "getRecords.intervalMs")
+ .key(PREFIX + "get_records.interval.ms")
.defaultValue(200L)
- .withAlternatives(OLD_PREFIX + "getRecords.intervalMs")
.markAdvanced()
.withDocumentation("Minimum interval in ms between two GetRecords API calls per shard.");
+ public static final ConfigProperty<Boolean> KINESIS_PERSIST_FETCH_RDD = ConfigProperty
+ .key(PREFIX + "persist.fetch.rdd")
+ .defaultValue(false)
+ .markAdvanced()
+ .withDocumentation("When enabled, the fetch RDD is persisted (MEMORY_AND_DISK) so it can be used "
+ + "both for record RDD and checkpoint summaries without recomputation. When false, the fetch "
+ + "is computed twice (once for checkpoint, once when the record RDD is consumed), which can "
+ + "cause duplicate records to be written. Set to true for correct, duplicate-free ingestion.");
+
Review Comment:
I set it to true; previously I don't know if we should use it or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r3034640289
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -38,116 +37,113 @@
description = "Configurations controlling the behavior of Kinesis source in Hudi Streamer.")
public class KinesisSourceConfig extends HoodieConfig {
- public static final String KINESIS_CHECKPOINT_TYPE_STRING = "string";
-
private static final String PREFIX = STREAMER_CONFIG_PREFIX + "source.kinesis.";
- private static final String OLD_PREFIX = DELTA_STREAMER_CONFIG_PREFIX + "source.kinesis.";
public static final ConfigProperty<String> KINESIS_STREAM_NAME = ConfigProperty
.key(PREFIX + "stream.name")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "stream.name")
.withDocumentation("Kinesis Data Streams stream name.");
public static final ConfigProperty<String> KINESIS_REGION = ConfigProperty
.key(PREFIX + "region")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "region")
+ .markAdvanced()
.withDocumentation("AWS region for the Kinesis stream (e.g., us-east-1).");
public static final ConfigProperty<String> KINESIS_ENDPOINT_URL = ConfigProperty
.key(PREFIX + "endpoint.url")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "endpoint.url")
.markAdvanced()
.withDocumentation("Custom endpoint URL for Kinesis (e.g., for localstack). "
+ "If not set, uses the default AWS endpoint for the region.");
public static final ConfigProperty<String> KINESIS_ACCESS_KEY = ConfigProperty
.key(PREFIX + "access.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "access.key")
.markAdvanced()
.withDocumentation("AWS access key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<String> KINESIS_SECRET_KEY = ConfigProperty
.key(PREFIX + "secret.key")
.noDefaultValue()
- .withAlternatives(OLD_PREFIX + "secret.key")
.markAdvanced()
.withDocumentation("AWS secret key for Kinesis. Used when connecting to custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential chain.");
public static final ConfigProperty<Long> MAX_EVENTS_FROM_KINESIS_SOURCE = ConfigProperty
- .key(STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
+ .key(PREFIX + "max.events")
.defaultValue(5000000L)
- .withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "kinesis.source.maxEvents")
.markAdvanced()
.withDocumentation("Maximum number of records obtained in each batch from Kinesis.");
- public static final ConfigProperty<Long> KINESIS_SOURCE_MIN_PARTITIONS = ConfigProperty
- .key(PREFIX + "minPartitions")
+ public static final ConfigProperty<Long> KINESIS_SOURCE_MANUAL_PARTITIONS = ConfigProperty
+ .key(PREFIX + "manual.partitions")
.defaultValue(0L)
- .withAlternatives(OLD_PREFIX + "minPartitions")
.markAdvanced()
- .withDocumentation("Desired minimum number of Spark partitions when reading from Kinesis. "
+ .withDocumentation("Desired number of Spark partitions when reading from Kinesis. "
+ "By default, Hudi has a 1-1 mapping of Kinesis shards to Spark partitions. "
- + "If set to a value greater than the number of shards, the result RDD will be repartitioned "
- + "to increase downstream parallelism. Use 0 for 1-1 mapping.");
+ + "If set to a value greater than 0, the result RDD will be repartitioned "
+ + "to increase/decrease downstream parallelism. Use 0 for 1-1 mapping.");
public static final ConfigProperty<Boolean> KINESIS_APPEND_OFFSETS = ConfigProperty
.key(PREFIX + "append.offsets")
- .defaultValue(false)
- .withAlternatives(OLD_PREFIX + "append.offsets")
+ .defaultValue(true)
Review Comment:
yes. we should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4185873752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064)
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 01d304e3b8813faff815fd1b8bd4520b344dcd1e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329235155
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.03%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.03% -0.86%
- Complexity 28537 28893 +356
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95617 +1295
- Misses 34996 37073 +2077
- Partials 7592 7848 +256
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.58% <0.00%> (-0.89%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [85 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326742549
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
}, {
"hash" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13640",
"triggerID" : "db19834cf6915d0a9e1674ed36c04c2a45d61545",
"triggerType" : "PUSH"
}, {
"hash" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642",
"triggerID" : "44f0050b30e808c35359269a6da7aa6bad6f926e",
"triggerType" : "PUSH"
}, {
"hash" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b477867a5091d193bc749516cc4fc0bc32d2a591",
"triggerType" : "PUSH"
}, {
"hash" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645",
"triggerID" : "cbaa9d827092f69c83d752739372f57c1c5a3250",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
* 44f0050b30e808c35359269a6da7aa6bad6f926e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13642)
* b477867a5091d193bc749516cc4fc0bc32d2a591 UNKNOWN
* cbaa9d827092f69c83d752739372f57c1c5a3250 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13645)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4327859927
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.18%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7af0201`](https://app.codecov.io/gh/apache/hudi/commit/7af0201256fc80be36c78bc2e56df221b876902a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (7af0201). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (7af0201) |
>|------|------|------|
>|spark-scala-tests|10|2|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 46.18% -22.72%
+ Complexity 28537 15406 -13131
=============================================
Files 2480 1978 -502
Lines 136910 109362 -27548
Branches 16679 13584 -3095
=============================================
- Hits 94322 50507 -43815
- Misses 34996 53520 +18524
+ Partials 7592 5335 -2257
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.02%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.94% <0.00%> (-12.33%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1622 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328165006
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.82%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a193e7e`](https://app.codecov.io/gh/apache/hudi/commit/a193e7e13b56a25526e18c3eec95cbdf3d16c79c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (a193e7e). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (a193e7e) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.82% -24.08%
+ Complexity 28537 8523 -20014
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28071 -66251
+ Misses 34996 31469 -3527
+ Partials 7592 3086 -4506
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (-0.01%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328735422
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.78%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.78% -24.12%
+ Complexity 28537 8508 -20029
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28045 -66277
+ Misses 34996 31509 -3487
+ Partials 7592 3072 -4520
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2078 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4328950817
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.57%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|1|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 46.57% -22.33%
+ Complexity 28537 15198 -13339
=============================================
Files 2480 1961 -519
Lines 136910 106850 -30060
Branches 16679 13016 -3663
=============================================
- Hits 94322 49763 -44559
- Misses 34996 51907 +16911
+ Partials 7592 5180 -2412
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.02% <0.00%> (-12.26%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1652 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329068984
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.60%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 11 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|8|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 59.60% -9.29%
+ Complexity 28537 21086 -7451
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15204 -1475
============================================
- Hits 94322 69683 -24639
- Misses 34996 40066 +5070
+ Partials 7592 7156 -436
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.59% <0.00%> (-17.88%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.41% <0.00%> (-0.87%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1278 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329046633
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.63%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (6f2372e). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (6f2372e) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|4|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.63% -21.26%
+ Complexity 28537 17001 -11536
=============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15204 -1475
=============================================
- Hits 94322 55687 -38635
- Misses 34996 55093 +20097
+ Partials 7592 6125 -1467
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.36% <0.00%> (-17.12%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <0.00%> (-12.73%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1572 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329127485
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.98%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 67.98% -0.91%
- Complexity 28537 28873 +336
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95548 +1226
- Misses 34996 37164 +2168
- Partials 7592 7826 +234
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.23% <0.00%> (-1.24%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [98 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4329170519
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `54.79675%` with `278 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.03%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f2372e`](https://app.codecov.io/gh/apache/hudi/commit/6f2372e844600c92ced164a44d8ef70ed742d004?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 22.97% | [109 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 46.46% | [100 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 77.20% | [26 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 68.03% -0.86%
- Complexity 28537 28893 +356
============================================
Files 2480 2518 +38
Lines 136910 140538 +3628
Branches 16679 17420 +741
============================================
+ Hits 94322 95615 +1293
- Misses 34996 37074 +2078
- Partials 7592 7849 +257
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.34% <54.79%> (-0.08%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.78% <ø> (-0.05%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.40% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.56% <0.00%> (-0.92%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.70% <0.00%> (-0.58%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.69% <0.00%> (-0.29%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `77.20% <77.20%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `46.46% <46.46%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `22.97% <22.97%> (ø)` | |
... and [86 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324736331
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.24%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 56.24% -12.66%
+ Complexity 28537 19305 -9232
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
=============================================
- Hits 94322 64255 -30067
- Misses 34996 43632 +8636
+ Partials 7592 6359 -1233
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.17% <0.00%> (-9.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.22% <0.00%> (-13.06%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1320 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324852030
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.61%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.61% -2.28%
+ Complexity 28537 23251 -5286
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 76106 -18216
+ Misses 34996 31359 -3637
+ Partials 7592 6781 -811
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.13% <0.00%> (-0.34%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.98% <0.00%> (-0.30%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [820 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325561920
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.57%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (db19834). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (db19834) |
>|------|------|------|
>|spark-scala-tests|10|1|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 46.57% -22.32%
+ Complexity 28537 15202 -13335
=============================================
Files 2480 1961 -519
Lines 136910 106983 -29927
Branches 16679 12991 -3688
=============================================
- Hits 94322 49828 -44494
- Misses 34996 51968 +16972
+ Partials 7592 5187 -2405
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.04% <0.00%> (-12.24%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1640 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4326682738
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.82%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cbaa9d8`](https://app.codecov.io/gh/apache/hudi/commit/cbaa9d827092f69c83d752739372f57c1c5a3250?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (cbaa9d8). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (cbaa9d8) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.82% -24.07%
+ Complexity 28537 8521 -20016
=============================================
Files 2480 1201 -1279
Lines 136910 62626 -74284
Branches 16679 6780 -9899
=============================================
- Hits 94322 28073 -66249
+ Misses 34996 31466 -3530
+ Partials 7592 3087 -4505
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2080 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324053262
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 54.00%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 54.00% -14.90%
+ Complexity 28537 12395 -16142
=============================================
Files 2480 1433 -1047
Lines 136910 71836 -65074
Branches 16679 8167 -8512
=============================================
- Hits 94322 38792 -55530
+ Misses 34996 29570 -5426
+ Partials 7592 3474 -4118
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1829 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324092042
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `615 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.57%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 0.00% | [198 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 0.00% | [148 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 0.00% | [136 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 0.00% | [107 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (282c1be). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (282c1be) |
>|------|------|------|
>|spark-scala-tests|10|5|
>|spark-java-tests|14|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 47.57% -21.32%
+ Complexity 28537 16568 -11969
=============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
=============================================
- Hits 94322 54356 -39966
- Misses 34996 54148 +19152
+ Partials 7592 5742 -1850
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.10% <0.00%> (-16.37%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.23% <0.00%> (-13.05%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [1547 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324195123
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.33%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`282c1be`](https://app.codecov.io/gh/apache/hudi/commit/282c1bea218876530b319f14dedd7a75a3fff469?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 66.33% -2.57%
+ Complexity 28537 23155 -5382
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14614 -2065
============================================
- Hits 94322 75782 -18540
+ Misses 34996 31693 -3303
+ Partials 7592 6771 -821
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.84% <ø> (+0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.45% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <0.00%> (-0.29%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.99% <0.00%> (-0.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.79% <69.91%> (-0.20%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [831 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324499903
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.85%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 44.85% -24.04%
+ Complexity 28537 8530 -20007
=============================================
Files 2480 1201 -1279
Lines 136910 62608 -74302
Branches 16679 6778 -9901
=============================================
- Hits 94322 28083 -66239
+ Misses 34996 31445 -3551
+ Partials 7592 3080 -4512
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2078 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324592455
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 54.00%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|0|
>|spark-java-tests|14|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
=============================================
- Coverage 68.89% 54.00% -14.89%
+ Complexity 28537 12398 -16139
=============================================
Files 2480 1433 -1047
Lines 136910 71836 -65074
Branches 16679 8167 -8512
=============================================
- Hits 94322 38797 -55525
+ Misses 34996 29568 -5428
+ Partials 7592 3471 -4121
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1828 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4324753088
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 61.87%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08ae177`](https://app.codecov.io/gh/apache/hudi/commit/08ae17785490db6a2b9432acbc3196517ed73845?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (436bd66) and HEAD (08ae177). Click for more details.
>
> <details><summary>HEAD has 13 uploads less than BASE</summary>
>
>| Flag | BASE (436bd66) | HEAD (08ae177) |
>|------|------|------|
>|spark-scala-tests|10|6|
>|spark-java-tests|14|6|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 61.87% -7.02%
+ Complexity 28537 21307 -7230
============================================
Files 2480 2026 -454
Lines 136910 114246 -22664
Branches 16679 14611 -2068
============================================
- Hits 94322 70694 -23628
- Misses 34996 36741 +1745
+ Partials 7592 6811 -781
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.85% <ø> (+0.03%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.46% <ø> (-0.07%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.17% <0.00%> (-9.31%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.39% <0.00%> (-0.89%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.15% <69.91%> (+0.16%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [1174 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325164970
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
}, {
"hash" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12513",
"triggerID" : "f6fbded79c4ca9b5f710d8eb9a33f164fc658c0a",
"triggerType" : "PUSH"
}, {
"hash" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12517",
"triggerID" : "8d77ec07f756f71fb3105c8a38008a995b05d793",
"triggerType" : "PUSH"
}, {
"hash" : "522d591c936caab680fd2585711c03cdc3414217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "522d591c936caab680fd2585711c03cdc3414217",
"triggerType" : "PUSH"
}, {
"hash" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12550",
"triggerID" : "7e41c5fd112cbd47e9554bbf27e6d23c2f6de149",
"triggerType" : "PUSH"
}, {
"hash" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12554",
"triggerID" : "9af3386e49cb4c09581bce9632b306b2b4ced5de",
"triggerType" : "PUSH"
}, {
"hash" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12557",
"triggerID" : "5aaba50d2b78079d344e8d39c6baf0e76da9f5ae",
"triggerType" : "PUSH"
}, {
"hash" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13062",
"triggerID" : "0980ea929e91ae9637a27f24f07f00bf2ed8db4d",
"triggerType" : "PUSH"
}, {
"hash" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13064",
"triggerID" : "3b7d4b5f88f228bbde8ac723fabaddc9f82a4b97",
"triggerType" : "PUSH"
}, {
"hash" : "9fefebc5833b73afb66193bd1c13779508792257",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fefebc5833b73afb66193bd1c13779508792257",
"triggerType" : "PUSH"
}, {
"hash" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13065",
"triggerID" : "01d304e3b8813faff815fd1b8bd4520b344dcd1e",
"triggerType" : "PUSH"
}, {
"hash" : "1834a270798e4610470f701dacdba46059e4c8af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1834a270798e4610470f701dacdba46059e4c8af",
"triggerType" : "PUSH"
}, {
"hash" : "282c1bea218876530b319f14dedd7a75a3fff469",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13624",
"triggerID" : "282c1bea218876530b319f14dedd7a75a3fff469",
"triggerType" : "PUSH"
}, {
"hash" : "08ae17785490db6a2b9432acbc3196517ed73845",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633",
"triggerID" : "08ae17785490db6a2b9432acbc3196517ed73845",
"triggerType" : "PUSH"
}, {
"hash" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4",
"triggerType" : "PUSH"
}, {
"hash" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638",
"triggerID" : "9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc",
"triggerType" : "PUSH"
}, {
"hash" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "58f96e64fea6709f559dd05f28c26cb13c9621eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 522d591c936caab680fd2585711c03cdc3414217 UNKNOWN
* 9fefebc5833b73afb66193bd1c13779508792257 UNKNOWN
* 1834a270798e4610470f701dacdba46059e4c8af UNKNOWN
* 08ae17785490db6a2b9432acbc3196517ed73845 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13633)
* 458f3f5d057ec0318ead1e6aae6bf9f0cb98eff4 UNKNOWN
* 9447e32dd661d32d57c6ca7f5e921d1c6ae5c3bc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13638)
* 58f96e64fea6709f559dd05f28c26cb13c9621eb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325716851
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.56%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.56% -3.33%
+ Complexity 28537 23377 -5160
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76648 -17674
+ Misses 34996 33284 -1712
+ Partials 7592 6973 -619
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.15% <0.00%> (-1.32%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.54% <0.00%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [878 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4325751456
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.91870%` with `185 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.72%. Comparing base ([`436bd66`](https://app.codecov.io/gh/apache/hudi/commit/436bd66834a5e86440bc4147bc35b003af9436ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`db19834`](https://app.codecov.io/gh/apache/hudi/commit/db19834cf6915d0a9e1674ed36c04c2a45d61545?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | 57.07% | [56 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | 61.76% | [42 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | 70.94% | [29 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | 84.61% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | 99.06% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18224 +/- ##
============================================
- Coverage 68.89% 65.72% -3.17%
+ Complexity 28537 23468 -5069
============================================
Files 2480 2047 -433
Lines 136910 116905 -20005
Branches 16679 15201 -1478
============================================
- Hits 94322 76835 -17487
+ Misses 34996 33084 -1912
+ Partials 7592 6986 -606
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.82% <ø> (+<0.01%)` | :arrow_up: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.41% <ø> (-0.12%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.42% <0.00%> (-1.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.61% <0.00%> (-0.67%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18224/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.09% <69.91%> (+0.11%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/hudi/utilities/config/KinesisSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FKinesisSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9LaW5lc2lzU291cmNlQ29uZmlnLmphdmE=) | `99.06% <99.06%> (ø)` | |
| [...utilities/sources/helpers/KinesisDeaggregator.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisDeaggregator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzRGVhZ2dyZWdhdG9yLmphdmE=) | `84.61% <84.61%> (ø)` | |
| [...ache/hudi/utilities/sources/JsonKinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FJsonKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbktpbmVzaXNTb3VyY2UuamF2YQ==) | `70.94% <70.94%> (ø)` | |
| [...g/apache/hudi/utilities/sources/KinesisSource.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FKinesisSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvS2luZXNpc1NvdXJjZS5qYXZh) | `61.76% <61.76%> (ø)` | |
| [...di/utilities/sources/helpers/KinesisOffsetGen.java](https://app.codecov.io/gh/apache/hudi/pull/18224?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FKinesisOffsetGen.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LaW5lc2lzT2Zmc2V0R2VuLmphdmE=) | `57.07% <57.07%> (ø)` | |
... and [865 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2872823645
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKinesisSource.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen;
+import org.apache.hudi.utilities.sources.helpers.KinesisReadConfig;
+import org.apache.hudi.utilities.streamer.DefaultStreamContext;
+import org.apache.hudi.utilities.streamer.StreamContext;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.model.Record;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.sources.helpers.KinesisOffsetGen.LOCALSTACK_END_SEQ_SENTINEL;
+
+/**
+ * Source to read JSON data from AWS Kinesis Data Streams using Spark.
+ */
+@Slf4j
+public class JsonKinesisSource extends KinesisSource<JavaRDD<String>> {
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ /** Result from reading a single shard in a partition. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchResult implements Serializable {
+ private final List<String> records;
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Metadata-only summary for checkpoint; avoids bringing records to driver. */
+ @AllArgsConstructor
+ @Getter
+ private static class ShardFetchSummary implements Serializable {
+ private final String shardId;
+ private final Option<String> lastSequenceNumber;
+ private final int recordCount;
+ private final boolean reachedEndOfShard;
+ }
+
+ /** Persisted fetch RDD - must be unpersisted in releaseResources to avoid memory leak. */
+ private transient org.apache.spark.api.java.JavaRDD<ShardFetchResult> persistedFetchRdd;
+ /** Record count from fetch, avoids redundant batch.count() Spark job. */
+ private long lastRecordCount;
+ /** Shard IDs where the executor observed nextShardIterator==null (end-of-shard reached). */
+ protected Set<String> shardsReachedEnd;
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ SchemaProvider schemaProvider, HoodieIngestionMetrics metrics) {
+ this(properties, sparkContext, sparkSession, metrics,
+ new DefaultStreamContext(schemaProvider, Option.empty()));
+ }
+
+ public JsonKinesisSource(TypedProperties properties, JavaSparkContext sparkContext, SparkSession sparkSession,
+ HoodieIngestionMetrics metrics, StreamContext streamContext) {
+ super(properties, sparkContext, sparkSession, SourceType.JSON, metrics,
+ new DefaultStreamContext(streamContext.getSchemaProvider(), streamContext.getSourceProfileSupplier()));
+ this.offsetGen = new KinesisOffsetGen(props);
+ }
+
+ @Override
+ protected JavaRDD<String> toBatch(KinesisOffsetGen.KinesisShardRange[] shardRanges) {
+ KinesisReadConfig readConfig = new KinesisReadConfig(
+ offsetGen.getStreamName(),
+ offsetGen.getRegion(),
+ offsetGen.getEndpointUrl().orElse(null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null),
+ getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null),
+ offsetGen.getStartingPosition(),
+ shouldAddOffsets,
+ getBooleanWithAltKeys(props, KinesisSourceConfig.KINESIS_ENABLE_DEAGGREGATION),
+ getIntWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_MAX_RECORDS),
+ getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_GET_RECORDS_INTERVAL_MS),
+ // Evenly set the max events per shard.
+ shardRanges.length > 0 ? Math.max(1, getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE) / shardRanges.length) : Long.MAX_VALUE);
+
+ JavaRDD<ShardFetchResult> fetchRdd = sparkContext.parallelize(
+ java.util.Arrays.asList(shardRanges), shardRanges.length)
+ .mapPartitions(shardRangeIt -> {
+ List<ShardFetchResult> results = new ArrayList<>();
+ try (KinesisClient client = createKinesisClientFromConfig(readConfig)) {
+ while (shardRangeIt.hasNext()) {
+ KinesisOffsetGen.KinesisShardRange range = shardRangeIt.next();
+ KinesisOffsetGen.ShardReadResult readResult = KinesisOffsetGen.readShardRecords(
+ client, readConfig.getStreamName(), range, readConfig.getStartingPosition(),
+ readConfig.getMaxRecordsPerRequest(), readConfig.getIntervalMs(), readConfig.getMaxRecordsPerShard(),
+ readConfig.isEnableDeaggregation());
+
+ List<String> recordStrings = new ArrayList<>();
+ for (Record r : readResult.getRecords()) {
+ String json = recordToJsonStatic(r, range.getShardId(), readConfig.isShouldAddOffsets());
+ if (json != null) {
+ recordStrings.add(json);
+ }
+ }
+ results.add(new ShardFetchResult(recordStrings, range.getShardId(),
+ readResult.getLastSequenceNumber(), readResult.isReachedEndOfShard()));
+ }
+ }
+ return results.iterator();
+ });
+
+ // Cache so we can both get records and checkpoint from the same RDD
+ fetchRdd.persist(org.apache.spark.storage.StorageLevel.MEMORY_AND_DISK());
+ persistedFetchRdd = fetchRdd;
+
+ // RDD for the shard data.
+ JavaRDD<String> recordRdd = fetchRdd.flatMap(r -> r.getRecords().iterator());
+ // Apply minimum partitions for downstream parallelism (similar to Kafka source)
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ if (minPartitions > 0 && minPartitions > shardRanges.length) {
+ int targetPartitions = (int) minPartitions;
+ log.info("Repartitioning from {} shards to {} partitions (minPartitions={})",
+ shardRanges.length, targetPartitions, minPartitions);
+ recordRdd = recordRdd.repartition(targetPartitions);
+ }
+
+ // Information for checkpoint generation.
+ List<ShardFetchSummary> summaries = fetchRdd
+ .map(r -> new ShardFetchSummary(r.getShardId(), r.getLastSequenceNumber(), r.getRecords().size(),
+ r.isReachedEndOfShard()))
+ .collect();
+ lastCheckpointData = buildCheckpointFromSummaries(summaries);
+ Set<String> reached = new HashSet<>();
+ for (ShardFetchSummary s : summaries) {
+ if (s.isReachedEndOfShard()) {
+ reached.add(s.getShardId());
+ }
+ }
+ shardsReachedEnd = reached;
+ lastRecordCount = summaries.stream().mapToLong(ShardFetchSummary::getRecordCount).sum();
+
+ return recordRdd;
+ }
+
+ private static KinesisClient createKinesisClientFromConfig(KinesisReadConfig config) {
+ software.amazon.awssdk.services.kinesis.KinesisClientBuilder builder =
+ KinesisClient.builder().region(software.amazon.awssdk.regions.Region.of(config.getRegion()));
+ if (config.getEndpointUrl() != null && !config.getEndpointUrl().isEmpty()) {
+ builder = builder.endpointOverride(java.net.URI.create(config.getEndpointUrl()));
+ }
+ if (config.getAccessKey() != null && !config.getAccessKey().isEmpty()
+ && config.getSecretKey() != null && !config.getSecretKey().isEmpty()) {
+ builder = builder.credentialsProvider(
+ software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create(
+ software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create(
+ config.getAccessKey(), config.getSecretKey())));
+ }
+ return builder.build();
+ }
+
+ private static String recordToJsonStatic(Record record, String shardId, boolean shouldAddOffsets) {
+ String dataStr = record.data().asUtf8String();
+ // Pure empty or null records in Kinesis is not meaningful.
+ if (dataStr == null || dataStr.trim().isEmpty()) {
+ return null;
+ }
+ if (shouldAddOffsets) {
+ try {
+ ObjectNode node = (ObjectNode) OBJECT_MAPPER.readTree(dataStr);
+ node.put("_hoodie_kinesis_source_sequence_number", record.sequenceNumber());
+ node.put("_hoodie_kinesis_source_shard_id", shardId);
+ node.put("_hoodie_kinesis_source_partition_key", record.partitionKey());
+ if (record.approximateArrivalTimestamp() != null) {
+ node.put("_hoodie_kinesis_source_timestamp",
+ record.approximateArrivalTimestamp().toEpochMilli());
+ }
+ return OBJECT_MAPPER.writeValueAsString(node);
+ } catch (Exception e) {
+ return dataStr;
+ }
+ }
+ return dataStr;
+ }
+
+ private Map<String, String> buildCheckpointFromSummaries(List<ShardFetchSummary> summaries) {
Review Comment:
yes; otherwise, Kinesis could re-read these old shards and produce duplicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913135972
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
Review Comment:
Not sure. We can try to test it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18224:
URL: https://github.com/apache/hudi/pull/18224#issuecomment-4015520277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c44c273366ff265822001c38f3a04f427085e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12129",
"triggerID" : "8c44c273366ff265822001c38f3a04f427085e58",
"triggerType" : "PUSH"
}, {
"hash" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ed32832bd77bce5d011ecbb3665bb9c829c5223",
"triggerType" : "PUSH"
}, {
"hash" : "fec6001976389d10a38feda0a413fb6043f23ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12134",
"triggerID" : "fec6001976389d10a38feda0a413fb6043f23ace",
"triggerType" : "PUSH"
}, {
"hash" : "d09affde11485fedf2aba0f93230da54669bf307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12139",
"triggerID" : "d09affde11485fedf2aba0f93230da54669bf307",
"triggerType" : "PUSH"
}, {
"hash" : "a517e2685c28a1101af008e73abfe04522484a17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12166",
"triggerID" : "a517e2685c28a1101af008e73abfe04522484a17",
"triggerType" : "PUSH"
}, {
"hash" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12172",
"triggerID" : "82ade47dcdb28b4fd8db26ff39980c5e72fca27b",
"triggerType" : "PUSH"
}, {
"hash" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12174",
"triggerID" : "01a34fe2172609ecab7c865f2e3052bc82e6cc9c",
"triggerType" : "PUSH"
}, {
"hash" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12182",
"triggerID" : "d43f8eb0a91cb06424b90e695b777ccc6d2ee5fe",
"triggerType" : "PUSH"
}, {
"hash" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12216",
"triggerID" : "917c414dba55734c588d3c43c4b0e16e137fb211",
"triggerType" : "PUSH"
}, {
"hash" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12227",
"triggerID" : "a28e9e73447bf984536783a76949fd1f98f531dc",
"triggerType" : "PUSH"
}, {
"hash" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483",
"triggerID" : "759d3cdbf5684af44f12fb2ec7ef40c38760ab23",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ed32832bd77bce5d011ecbb3665bb9c829c5223 UNKNOWN
* 759d3cdbf5684af44f12fb2ec7ef40c38760ab23 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Create JsonKinesisSource [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #18224:
URL: https://github.com/apache/hudi/pull/18224#discussion_r2913360225
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisOffsetGen.java:
##########
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.table.checkpoint.Checkpoint;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.exception.HoodieReadFromSourceException;
+import org.apache.hudi.utilities.config.KinesisSourceConfig;
+import org.apache.hudi.utilities.ingestion.HoodieIngestionMetrics;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.kinesis.KinesisClient;
+import software.amazon.awssdk.services.kinesis.KinesisClientBuilder;
+import software.amazon.awssdk.services.kinesis.model.ExpiredIteratorException;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsRequest;
+import software.amazon.awssdk.services.kinesis.model.GetRecordsResponse;
+import software.amazon.awssdk.services.kinesis.model.GetShardIteratorRequest;
+import software.amazon.awssdk.services.kinesis.model.InvalidArgumentException;
+import software.amazon.awssdk.services.kinesis.model.LimitExceededException;
+import software.amazon.awssdk.services.kinesis.model.ListShardsRequest;
+import software.amazon.awssdk.services.kinesis.model.ListShardsResponse;
+import software.amazon.awssdk.services.kinesis.model.ProvisionedThroughputExceededException;
+import software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.kinesis.model.Record;
+import software.amazon.awssdk.services.kinesis.model.Shard;
+import software.amazon.awssdk.services.kinesis.model.ShardIteratorType;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties;
+import static org.apache.hudi.common.util.ConfigUtils.getBooleanWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Helper for reading from Kinesis Data Streams and managing checkpoints.
+ * Checkpoint format: streamName,shardId:sequenceNumber,shardId:sequenceNumber,...
+ */
+@Slf4j
+@Getter
+public class KinesisOffsetGen {
+
+ public static class CheckpointUtils {
+ /** Separator between lastSeq and endSeq for closed shards. Seq numbers are numeric, so this is safe. */
+ private static final String END_SEQ_SEPARATOR = "|";
+ /**
+ * Kinesis checkpoint pattern.
+ * Format: streamName,shardId:lastSeq,shardId:lastSeq|endSeq,...
+ * For closed shards we store lastSeq|endSeq so we can detect data loss when shard expires.
+ */
+ private static final Pattern PATTERN = Pattern.compile(".*,.*:.*");
+
+ /**
+ * Parse checkpoint string to shardId -> value map. Value is lastSeq or lastSeq|endSeq for closed shards.
+ */
+ public static Map<String, String> strToOffsets(String checkpointStr) {
+ Map<String, String> offsetMap = new HashMap<>();
+ String[] splits = checkpointStr.split(",");
+ for (int i = 1; i < splits.length; i++) {
+ String part = splits[i];
+ int colonIdx = part.indexOf(':');
+ if (colonIdx > 0 && colonIdx < part.length() - 1) {
+ String shardId = part.substring(0, colonIdx);
+ String value = part.substring(colonIdx + 1);
+ offsetMap.put(shardId, value);
+ }
+ }
+ return offsetMap;
+ }
+
+ /**
+ * Extract lastSeq from checkpoint value (which may be "lastSeq" or "lastSeq|endSeq").
+ */
+ public static String getLastSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return value;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 ? value.substring(0, sep) : value;
+ }
+
+ /**
+ * Extract endSeq from checkpoint value if present. Returns null for open shards.
+ */
+ public static String getEndSeqFromValue(String value) {
+ if (value == null || value.isEmpty()) {
+ return null;
+ }
+ int sep = value.indexOf(END_SEQ_SEPARATOR);
+ return sep >= 0 && sep < value.length() - 1 ? value.substring(sep + 1) : null;
+ }
+
+ /**
+ * Build checkpoint value: "lastSeq" or "lastSeq|endSeq" when endSeq is present (closed shards).
+ */
+ public static String buildCheckpointValue(String lastSeq, String endSeq) {
+ if (endSeq != null && !endSeq.isEmpty()) {
+ return lastSeq + END_SEQ_SEPARATOR + endSeq;
+ }
+ return lastSeq;
+ }
+
+ /**
+ * String representation of checkpoint.
+ * Format: streamName,shardId:value,shardId:value,... where value is lastSeq or lastSeq|endSeq.
+ */
+ public static String offsetsToStr(String streamName, Map<String, String> shardToValue) {
+ String parts = shardToValue.entrySet().stream()
+ .sorted(Map.Entry.comparingByKey())
+ .map(e -> e.getKey() + ":" + e.getValue())
+ .collect(Collectors.joining(","));
+ return streamName + "," + parts;
+ }
+
+ public static boolean checkStreamCheckpoint(Option<String> lastCheckpointStr) {
+ return lastCheckpointStr.isPresent() && PATTERN.matcher(lastCheckpointStr.get()).matches();
+ }
+ }
+
+ /** LocalStack returns Long.MAX_VALUE for closed shards' endingSequenceNumber; real AWS returns actual value. */
+ public static final String LOCALSTACK_END_SEQ_SENTINEL = "9223372036854775807";
+
+ /**
+ * Represents a shard to read from, with optional starting sequence number.
+ * For closed shards, endingSequenceNumber is set so we can store it in the checkpoint
+ * and later detect data loss when the shard expires.
+ */
+ @AllArgsConstructor
+ @Getter
+ public static class KinesisShardRange implements java.io.Serializable {
+ private final String shardId;
+ /** If empty, use TRIM_HORIZON or LATEST based on config. */
+ private final Option<String> startingSequenceNumber;
+ /** For closed shards: the shard's ending sequence number. Empty for open shards. */
+ private final Option<String> endingSequenceNumber;
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum) {
+ return new KinesisShardRange(shardId, seqNum, Option.empty());
+ }
+
+ public static KinesisShardRange of(String shardId, Option<String> seqNum, Option<String> endSeq) {
+ return new KinesisShardRange(shardId, seqNum, endSeq);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * Uses conservative default (useLatestWhenNoCheckpoint=false).
+ */
+ public boolean hasUnreadRecords() {
+ return hasUnreadRecords(false);
+ }
+
+ /**
+ * Returns true if this range may have unread records, false if we can definitively determine it has none.
+ * <ul>
+ * <li>Open shard: always true (may have new records)</li>
+ * <li>Closed shard, lastSeq >= endSeq: false (fully consumed)</li>
+ * <li>Closed shard, no checkpoint and useLatest: false (at LATEST tip, closed shard has no records)</li>
+ * <li>Closed shard with LocalStack endSeq sentinel and lastSeq equals sentinel: false (fully consumed)</li>
+ * <li>Otherwise: true (may have unread records or cannot definitively say)</li>
+ * </ul>
+ *
+ * @param useLatestWhenNoCheckpoint when startingSequenceNumber is empty, true means we use LATEST
+ * (start at tip); for a closed shard there are no records to read
+ */
+ public boolean hasUnreadRecords(boolean useLatestWhenNoCheckpoint) {
+ String lastSeq = startingSequenceNumber.orElse(null);
+ String endSeq = endingSequenceNumber.orElse(null);
+
+ // Open shard: may have records
+ if (endSeq == null || endSeq.isEmpty()) {
+ return true;
+ }
+
+ // Closed shard with no checkpoint
+ if (lastSeq == null || lastSeq.isEmpty()) {
+ return !useLatestWhenNoCheckpoint;
+ }
+
+ // Closed shard: lastSeq >= endSeq means fully consumed
+ if (lastSeq.compareTo(endSeq) >= 0) {
+ return false;
+ }
+
+ // LocalStack sentinel: when lastSeq equals sentinel, we've fully consumed
+ if (LOCALSTACK_END_SEQ_SENTINEL.equals(endSeq) && LOCALSTACK_END_SEQ_SENTINEL.equals(lastSeq)) {
+ return false;
+ }
+
+ // lastSeq < endSeq or ambiguous (e.g. LocalStack sentinel): may have unread records
+ return true;
+ }
+ }
+
+ private final String streamName;
+ private final String region;
+ private final Option<String> endpointUrl;
+ private final KinesisSourceConfig.KinesisStartingPosition startingPosition;
+ private final TypedProperties props;
+
+ public KinesisOffsetGen(TypedProperties props) {
+ this.props = props;
+ checkRequiredConfigProperties(props,
+ Arrays.asList(KinesisSourceConfig.KINESIS_STREAM_NAME, KinesisSourceConfig.KINESIS_REGION));
+ this.streamName = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STREAM_NAME);
+ this.region = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_REGION);
+ this.endpointUrl = Option.ofNullable(getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ENDPOINT_URL, null));
+ String posStr = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_STARTING_POSITION, true);
+ String normalized = posStr.toUpperCase().replace("EARLIEST", "TRIM_HORIZON");
+ this.startingPosition = KinesisSourceConfig.KinesisStartingPosition.valueOf(normalized);
+ }
+
+ public KinesisClient createKinesisClient() {
+ KinesisClientBuilder builder = KinesisClient.builder().region(Region.of(region));
+ if (endpointUrl.isPresent()) {
+ builder = builder.endpointOverride(URI.create(endpointUrl.get()));
+ }
+ String accessKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_ACCESS_KEY, null);
+ String secretKey = getStringWithAltKeys(props, KinesisSourceConfig.KINESIS_SECRET_KEY, null);
+ if (accessKey != null && !accessKey.isEmpty() && secretKey != null && !secretKey.isEmpty()) {
+ builder = builder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)));
+ }
+ return builder.build();
+ }
+
+ /**
+ * List all active shards for the stream.
+ * Note: AWS API disallows streamName and nextToken in the same request.
+ */
+ public List<Shard> listShards(KinesisClient client) {
+ List<Shard> allShards = new ArrayList<>();
+ String nextToken = null;
+ do {
+ ListShardsRequest request = nextToken != null
+ ? ListShardsRequest.builder().nextToken(nextToken).build()
+ : ListShardsRequest.builder().streamName(streamName).build();
+ ListShardsResponse response;
+ try {
+ response = client.listShards(request);
+ } catch (ResourceNotFoundException e) {
+ throw new HoodieReadFromSourceException("Kinesis stream " + streamName + " not found", e);
+ } catch (ProvisionedThroughputExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis throughput exceeded listing shards for " + streamName, e);
+ } catch (LimitExceededException e) {
+ throw new HoodieReadFromSourceException("Kinesis limit exceeded listing shards: " + e.getMessage(), e);
+ }
+ allShards.addAll(response.shards());
+ nextToken = response.nextToken();
+ } while (nextToken != null);
+
+ // Include both open and closed shards. Closed shards (e.g., from resharding) may still contain
+ // unread records within the retention period. GetRecords works on closed shards until all data
+ // is consumed, at which point NextShardIterator returns null.
+ long openCount = allShards.stream()
+ .filter(s -> s.sequenceNumberRange() != null && s.sequenceNumberRange().endingSequenceNumber() == null)
+ .count();
+ log.info("Found {} shards for stream {} ({} open, {} closed)",
+ allShards.size(), streamName, openCount, allShards.size() - openCount);
+ return allShards;
+ }
+
+ /**
+ * Get shard ranges to read, based on checkpoint and limits.
+ */
+ public KinesisShardRange[] getNextShardRanges(Option<Checkpoint> lastCheckpoint,
+ long sourceLimit,
+ HoodieIngestionMetrics metrics) {
+ long maxEvents = getLongWithAltKeys(props, KinesisSourceConfig.MAX_EVENTS_FROM_KINESIS_SOURCE);
+ long numEvents = sourceLimit == Long.MAX_VALUE ? maxEvents : Math.min(sourceLimit, maxEvents);
+ long minPartitions = getLongWithAltKeys(props, KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS);
+ log.info("getNextShardRanges set config {} to {}", KinesisSourceConfig.KINESIS_SOURCE_MIN_PARTITIONS.key(), minPartitions);
+
+ try (KinesisClient client = createKinesisClient()) {
+ // List all open and closed shards.
+ List<Shard> shards = listShards(client);
+ if (shards.isEmpty()) {
+ return new KinesisShardRange[0];
+ }
+
+ Map<String, String> fromSequenceNumbers = new HashMap<>();
+ Option<String> lastCheckpointStr = lastCheckpoint.isPresent()
+ ? Option.of(lastCheckpoint.get().getCheckpointKey()) : Option.empty();
+
+ // CASE: last checkpoint exists.
+ if (lastCheckpointStr.isPresent() && CheckpointUtils.checkStreamCheckpoint(lastCheckpointStr)) {
+ Map<String, String> checkpointOffsets = CheckpointUtils.strToOffsets(lastCheckpointStr.get());
+ if (!checkpointOffsets.isEmpty() && lastCheckpointStr.get().startsWith(streamName + ",")) {
Review Comment:
Merge into isValidStreamCheckpoint function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org