You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/27 04:20:01 UTC
[PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
yihua opened a new pull request, #20086:
URL: https://github.com/apache/hudi/pull/20086
### Describe the issue this Pull Request addresses
closes #20082
part of #20064
Stacked on #20078 (and #20077), review those first. Until they merge this PR shows their commits plus one commit of this change.
In the Hudi Trino connector, every split with log files built a `HoodieTableMetaClient` on the worker. That read `hoodie.properties` and the index definitions (neither goes through the Trino file system cache), resolved the table schema from the timeline and the latest commit metadata unless column name casing resolution was on, and for tables before version 8 loaded the timeline again to check log blocks against the committed instants. A query over N such file groups did this N times.
### Summary and Changelog
Workers read file groups from table state that the coordinator ships on the table handle, so they no longer read table metadata from storage. `HudiTableHandle` carries the table config (without the create schema) and, for merge-on-read tables before version 8, the committed instants up to the query's latest commit (new `HudiCommittedInstants`). `HudiMetadata` ships the table schema for every merge-on-read handle. `HudiPageSourceProvider` builds a `FileGroupReaderTableState` from the handle and passes it with the split's storage to the file group reader.
Tests: `TestHudiWorkerTableMetadataAccess` asserts through file system tracing spans that workers read log files but nothing under `.hoodie`, for table versions 6 and 8 (it fails on the parent commit), and checks what the coordinator puts on the handle. `TestHudiTableHandle` covers the JSON round trip and the bounded capture of committed instants.
### Impact
No worker-side `.hoodie` I/O for splits with log files, down from 3 to 4 metadata reads per split. For merge-on-read handles the coordinator resolves the table schema once per query, even when no split has log files. The handle grows to about 2.5 to 3.2 KB for typical merge-on-read tables (about 9 KB for a 60-column table), plus 20 to 40 bytes per active commit instant for version 6 and 7 merge-on-read tables.
### Risk Level
low. The read path gets the same table config, schema and committed-instant semantics as before, captured once on the coordinator. The full hudi-trino suite passes, including the version 6 and 8 merge-on-read smoke and merge-semantics tests.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #20086:
URL: https://github.com/apache/hudi/pull/20086#discussion_r4178153498
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiWorkerTableMetadataAccess.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableSet;
+import io.opentelemetry.sdk.trace.data.SpanData;
+import io.trino.metadata.Metadata;
+import io.trino.metadata.QualifiedObjectName;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer.TestingTable;
+import io.trino.testing.AbstractTestQueryFramework;
+import io.trino.testing.DistributedQueryRunner;
+import io.trino.testing.QueryRunner;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableMap.toImmutableMap;
+import static io.trino.filesystem.tracing.FileSystemAttributes.FILE_LOCATION;
+import static io.trino.testing.TransactionBuilder.transaction;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
+
+/**
+ * Asserts that reading file groups with log files on a worker does not touch the table metadata under
+ * {@code .hoodie}: the table config, schema and timeline come from the table handle built on the coordinator.
+ */
+@Execution(SAME_THREAD)
+final class TestHudiWorkerTableMetadataAccess
+ extends AbstractTestQueryFramework
+{
+ // Spans the task executors open around split processing; file system spans under them come from the page source path
+ private static final Set<String> SPLIT_PROCESSING_SPANS = ImmutableSet.of("process", "split");
+
+ @Override
+ protected QueryRunner createQueryRunner()
+ throws Exception
+ {
+ return HudiQueryRunner.builder()
+ .addConnectorProperty("fs.cache.enabled", "false")
Review Comment:
**nit:** `hudi.metadata.cache.enabled` defaults to true, so a cached `.hoodie` timeline read may leave no `FILE_LOCATION` span and slip past the `noneMatch("/.hoodie")` check. `TestHudiNoCacheFileOperations` turns off both caches for that reason. Feel free to ignore, but could we turn it off here too?
```suggestion
.addConnectorProperty("fs.cache.enabled", "false")
.addConnectorProperty("hudi.metadata.cache.enabled", "false")
```
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiTableHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import io.airlift.json.JsonCodec;
+import io.trino.spi.predicate.TupleDomain;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.read.FileGroupReaderTableState;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.InstantGenerator;
+import org.apache.hudi.common.table.timeline.TimelineLayout;
+import org.apache.hudi.storage.StoragePath;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+
+import static io.airlift.json.JsonCodec.jsonCodec;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+final class TestHudiTableHandle
+{
+ private static final JsonCodec<HudiTableHandle> CODEC = jsonCodec(HudiTableHandle.class);
+ private static final Map<String, String> TABLE_CONFIG = ImmutableMap.of(
+ "hoodie.table.name", "test_table",
+ "hoodie.table.type", "MERGE_ON_READ",
+ "hoodie.table.version", "6");
+
+ @Test
+ void testWorkerTableStateFromJson()
+ {
+ HudiCommittedInstants committedInstants = new HudiCommittedInstants(
+ ImmutableList.of("20240201000000000", "20240401000000000"),
+ ImmutableList.of("20240301000000000"),
+ Optional.of("20240201000000000"));
+ HudiTableHandle handle = CODEC.fromJson(CODEC.toJson(createTableHandle(TABLE_CONFIG, Optional.of(committedInstants))));
+
+ assertThat(handle.getTableConfig()).isEqualTo(TABLE_CONFIG);
+ assertThat(handle.getCommittedInstants()).contains(committedInstants);
+ FileGroupReaderTableState tableState = handle.getFileGroupReaderTableState();
+ assertThat(tableState.getBasePath()).isEqualTo(new StoragePath("/test/path"));
+ assertThat(tableState.getTableConfig().getTableName()).isEqualTo("test_table");
+ assertThat(tableState.getTableConfig().getTableVersion()).isEqualTo(HoodieTableVersion.SIX);
+ // Before the timeline start counts as committed (archived), inflight and unknown instants do not
+ assertThat(tableState.isCommitted("20240101000000000")).isTrue();
+ assertThat(tableState.isCommitted("20240401000000000")).isTrue();
+ assertThat(tableState.isCommitted("20240301000000000")).isFalse();
+ assertThat(tableState.isCommitted("20240215000000000")).isFalse();
+ }
+
+ @Test
+ void testWorkerTableStateRequiresTableConfig()
+ {
+ HudiTableHandle handle = CODEC.fromJson(CODEC.toJson(createTableHandle(ImmutableMap.of(), Optional.empty())));
+
+ assertThatThrownBy(handle::getFileGroupReaderTableState)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessageContaining("carries no table config");
+ }
+
+ @Test
+ void testCaptureCommittedInstantsUpToLatestCommit()
+ {
+ InstantGenerator instantGenerator = TimelineLayout.TIMELINE_LAYOUT_V1.getInstantGenerator();
+ List<HoodieInstant> instants = ImmutableList.of(
+ instantGenerator.createNewInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "20240201000000000"),
+ instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "20240301000000000"),
+ instantGenerator.createNewInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "20240401000000000"),
+ instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "20240501000000000"),
+ instantGenerator.createNewInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "20240601000000000"));
+ HoodieTimeline timeline = TimelineLayout.TIMELINE_LAYOUT_V1.getTimelineFactory().createDefaultTimeline(instants.stream(), null);
+
+ assertThat(HudiCommittedInstants.capture(timeline, "20240401000000000")).isEqualTo(new HudiCommittedInstants(
+ ImmutableList.of("20240201000000000", "20240401000000000"),
+ ImmutableList.of("20240301000000000"),
+ Optional.of("20240201000000000")));
+ }
+
+ private static HudiTableHandle createTableHandle(Map<String, String> tableConfig, Optional<HudiCommittedInstants> committedInstants)
Review Comment:
**nit:** This is the fourth copy of the 13-argument handle construction (also in `TestHudiSplitSource`, `TestHudiSplitFactory` and `TestHudiPartitionInfoLoader`), and this commit had to append `ImmutableMap.of(), Optional.empty()` to the other three. Feel free to ignore, but could we add a `HudiTestUtils.testTableHandle(...)` factory so the next handle field is a one-line change?
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiWorkerTableMetadataAccess.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableSet;
+import io.opentelemetry.sdk.trace.data.SpanData;
+import io.trino.metadata.Metadata;
+import io.trino.metadata.QualifiedObjectName;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer.TestingTable;
+import io.trino.testing.AbstractTestQueryFramework;
+import io.trino.testing.DistributedQueryRunner;
+import io.trino.testing.QueryRunner;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableMap.toImmutableMap;
+import static io.trino.filesystem.tracing.FileSystemAttributes.FILE_LOCATION;
+import static io.trino.testing.TransactionBuilder.transaction;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
+
+/**
+ * Asserts that reading file groups with log files on a worker does not touch the table metadata under
+ * {@code .hoodie}: the table config, schema and timeline come from the table handle built on the coordinator.
+ */
+@Execution(SAME_THREAD)
+final class TestHudiWorkerTableMetadataAccess
+ extends AbstractTestQueryFramework
+{
+ // Spans the task executors open around split processing; file system spans under them come from the page source path
+ private static final Set<String> SPLIT_PROCESSING_SPANS = ImmutableSet.of("process", "split");
+
+ @Override
+ protected QueryRunner createQueryRunner()
Review Comment:
**nit:** This starts one more query runner over every resource fixture, with nearly the same setup as `TestHudiNoCacheFileOperations` (fs cache off, table statistics off, `ResourceHudiTablesInitializer`). Feel free to ignore, but could these two methods live there instead, or could a comment say why they need their own runner with a separate worker node?
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiWorkerTableMetadataAccess.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableSet;
+import io.opentelemetry.sdk.trace.data.SpanData;
+import io.trino.metadata.Metadata;
+import io.trino.metadata.QualifiedObjectName;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer.TestingTable;
+import io.trino.testing.AbstractTestQueryFramework;
+import io.trino.testing.DistributedQueryRunner;
+import io.trino.testing.QueryRunner;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableMap.toImmutableMap;
+import static io.trino.filesystem.tracing.FileSystemAttributes.FILE_LOCATION;
+import static io.trino.testing.TransactionBuilder.transaction;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
+
+/**
+ * Asserts that reading file groups with log files on a worker does not touch the table metadata under
+ * {@code .hoodie}: the table config, schema and timeline come from the table handle built on the coordinator.
+ */
+@Execution(SAME_THREAD)
+final class TestHudiWorkerTableMetadataAccess
+ extends AbstractTestQueryFramework
+{
+ // Spans the task executors open around split processing; file system spans under them come from the page source path
+ private static final Set<String> SPLIT_PROCESSING_SPANS = ImmutableSet.of("process", "split");
+
+ @Override
+ protected QueryRunner createQueryRunner()
+ throws Exception
+ {
+ return HudiQueryRunner.builder()
+ .addConnectorProperty("fs.cache.enabled", "false")
+ .addConnectorProperty("hudi.table-statistics-enabled", "false")
+ .setDataLoader(new ResourceHudiTablesInitializer())
+ .setWorkerCount(1)
+ .build();
+ }
+
+ @ParameterizedTest
+ @EnumSource(value = TestingTable.class, names = {"HUDI_COMPREHENSIVE_TYPES_V6_MOR", "HUDI_COMPREHENSIVE_TYPES_V8_MOR"})
+ void testLogFileSplitsDoNotReadTableMetadata(TestingTable table)
+ {
+ DistributedQueryRunner queryRunner = getDistributedQueryRunner();
+ queryRunner.executeWithPlan(getSession(), "SELECT * FROM " + table.getRtTableName());
+
+ String tableDirectory = "/" + table.getTableName() + "/";
+ List<String> splitFileLocations = splitProcessingFileLocations(queryRunner.getSpans()).stream()
+ .filter(location -> location.contains(tableDirectory))
+ .collect(toImmutableList());
+ assertThat(splitFileLocations).anyMatch(location -> location.contains(".log."));
+ assertThat(splitFileLocations).noneMatch(location -> location.contains("/.hoodie"));
Review Comment:
**nit:** `FileOperationUtils.FileType.fromFilePath` already classifies `.log` and `.hoodie` paths. Feel free to ignore, but could we map the locations through it and assert `contains(LOG)` plus none of the metadata types, so a failure names which metadata file was read? `fromFilePath` throws on unknown paths, so the table-directory filter would stay first.
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiWorkerTableMetadataAccess.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableSet;
+import io.opentelemetry.sdk.trace.data.SpanData;
+import io.trino.metadata.Metadata;
+import io.trino.metadata.QualifiedObjectName;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer;
+import io.trino.plugin.hudi.testing.ResourceHudiTablesInitializer.TestingTable;
+import io.trino.testing.AbstractTestQueryFramework;
+import io.trino.testing.DistributedQueryRunner;
+import io.trino.testing.QueryRunner;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableMap.toImmutableMap;
+import static io.trino.filesystem.tracing.FileSystemAttributes.FILE_LOCATION;
+import static io.trino.testing.TransactionBuilder.transaction;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
+
+/**
+ * Asserts that reading file groups with log files on a worker does not touch the table metadata under
+ * {@code .hoodie}: the table config, schema and timeline come from the table handle built on the coordinator.
+ */
+@Execution(SAME_THREAD)
+final class TestHudiWorkerTableMetadataAccess
+ extends AbstractTestQueryFramework
+{
+ // Spans the task executors open around split processing; file system spans under them come from the page source path
+ private static final Set<String> SPLIT_PROCESSING_SPANS = ImmutableSet.of("process", "split");
+
+ @Override
+ protected QueryRunner createQueryRunner()
+ throws Exception
+ {
+ return HudiQueryRunner.builder()
+ .addConnectorProperty("fs.cache.enabled", "false")
+ .addConnectorProperty("hudi.table-statistics-enabled", "false")
+ .setDataLoader(new ResourceHudiTablesInitializer())
+ .setWorkerCount(1)
+ .build();
+ }
+
+ @ParameterizedTest
+ @EnumSource(value = TestingTable.class, names = {"HUDI_COMPREHENSIVE_TYPES_V6_MOR", "HUDI_COMPREHENSIVE_TYPES_V8_MOR"})
+ void testLogFileSplitsDoNotReadTableMetadata(TestingTable table)
+ {
+ DistributedQueryRunner queryRunner = getDistributedQueryRunner();
+ queryRunner.executeWithPlan(getSession(), "SELECT * FROM " + table.getRtTableName());
+
+ String tableDirectory = "/" + table.getTableName() + "/";
+ List<String> splitFileLocations = splitProcessingFileLocations(queryRunner.getSpans()).stream()
+ .filter(location -> location.contains(tableDirectory))
+ .collect(toImmutableList());
+ assertThat(splitFileLocations).anyMatch(location -> location.contains(".log."));
+ assertThat(splitFileLocations).noneMatch(location -> location.contains("/.hoodie"));
+ }
+
+ @ParameterizedTest
+ @EnumSource(value = TestingTable.class, names = {"HUDI_COMPREHENSIVE_TYPES_V6_MOR", "HUDI_COMPREHENSIVE_TYPES_V8_MOR"})
+ void testTableHandleCarriesWorkerTableState(TestingTable table)
+ {
+ HudiTableHandle handle = coordinatorTableHandle(table.getRtTableName());
+
+ assertThat(handle.getTableSchemaStr()).isNotEmpty();
+ assertThat(handle.getTableConfig())
+ .containsKey(HoodieTableConfig.VERSION.key())
+ .doesNotContainKey(HoodieTableConfig.CREATE_SCHEMA.key());
+ assertThat(handle.getCommittedInstants().isPresent())
+ .isEqualTo(handle.getFileGroupReaderTableState().getTableConfig().getTableVersion().lesserThan(HoodieTableVersion.EIGHT));
Review Comment:
**nit:** The expected value is derived from the handle under test, and only presence is checked. Feel free to ignore, but could we pass the expectation as a parameter (V6 present, V8 empty) and, for V6, assert that `completedInstants()` equals the fixture's delta commit times and `inflightInstants()` is empty?
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/TestHudiTableHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.trino.plugin.hudi;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import io.airlift.json.JsonCodec;
+import io.trino.spi.predicate.TupleDomain;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.read.FileGroupReaderTableState;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.InstantGenerator;
+import org.apache.hudi.common.table.timeline.TimelineLayout;
+import org.apache.hudi.storage.StoragePath;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+
+import static io.airlift.json.JsonCodec.jsonCodec;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+final class TestHudiTableHandle
+{
+ private static final JsonCodec<HudiTableHandle> CODEC = jsonCodec(HudiTableHandle.class);
+ private static final Map<String, String> TABLE_CONFIG = ImmutableMap.of(
+ "hoodie.table.name", "test_table",
+ "hoodie.table.type", "MERGE_ON_READ",
+ "hoodie.table.version", "6");
+
+ @Test
+ void testWorkerTableStateFromJson()
+ {
+ HudiCommittedInstants committedInstants = new HudiCommittedInstants(
+ ImmutableList.of("20240201000000000", "20240401000000000"),
+ ImmutableList.of("20240301000000000"),
+ Optional.of("20240201000000000"));
+ HudiTableHandle handle = CODEC.fromJson(CODEC.toJson(createTableHandle(TABLE_CONFIG, Optional.of(committedInstants))));
+
+ assertThat(handle.getTableConfig()).isEqualTo(TABLE_CONFIG);
+ assertThat(handle.getCommittedInstants()).contains(committedInstants);
+ FileGroupReaderTableState tableState = handle.getFileGroupReaderTableState();
+ assertThat(tableState.getBasePath()).isEqualTo(new StoragePath("/test/path"));
+ assertThat(tableState.getTableConfig().getTableName()).isEqualTo("test_table");
+ assertThat(tableState.getTableConfig().getTableVersion()).isEqualTo(HoodieTableVersion.SIX);
+ // Before the timeline start counts as committed (archived), inflight and unknown instants do not
+ assertThat(tableState.isCommitted("20240101000000000")).isTrue();
+ assertThat(tableState.isCommitted("20240401000000000")).isTrue();
+ assertThat(tableState.isCommitted("20240301000000000")).isFalse();
+ assertThat(tableState.isCommitted("20240215000000000")).isFalse();
Review Comment:
**nit:** These four checks repeat `TestFileGroupReaderTableState` from #20078 with the same instants (lines 84-87 and 146-147). The new part here is the JSON round trip, which line 57 already asserts. Feel free to ignore, but could we keep one, e.g. the inflight `20240301000000000` not being committed, as the wiring check and leave the semantics to the hudi-common test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20086:
URL: https://github.com/apache/hudi/pull/20086#issuecomment-5852615021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f9c3880c9afb7ee1a8254fea671e91780257739 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20086:
URL: https://github.com/apache/hudi/pull/20086#issuecomment-5852650283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17300",
"triggerID" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f9c3880c9afb7ee1a8254fea671e91780257739 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17300)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20086:
URL: https://github.com/apache/hudi/pull/20086#issuecomment-5853340194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17300",
"triggerID" : "5f9c3880c9afb7ee1a8254fea671e91780257739",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f9c3880c9afb7ee1a8254fea671e91780257739 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17300)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #20086:
URL: https://github.com/apache/hudi/pull/20086#discussion_r4178137487
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/HudiTableHandle.java:
##########
@@ -248,6 +346,28 @@ public HoodieSchema getTableSchema()
return hudiTableSchema.map(Lazy::get).orElse(null);
}
+ @JsonProperty
+ public Map<String, String> getTableConfig()
+ {
+ return lazyTableConfig.get();
+ }
Review Comment:
`getTableConfig()` is serialized for every handle, so COW and _ro tables also shop the full **hoodie.properties** map.
`captureCommittedInstants` already skips non-MOR tables, can we skip the table config too?
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/HudiTableHandle.java:
##########
Review Comment:
MOR handles will always send schema to workers as a string, the workers will parse it back with `HoodieSchema#parse`.
Using this will check field defaults iirc. before, with column-name casing off, workers used the resolver's schema object directly. The resolver could return a log-block schema that was parsed without default checking.
Would it be safer for the worker to use `HoodieSchema.parse(tableSchemaStr, false)` instead?
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/HudiTableHandle.java:
##########
@@ -151,6 +207,48 @@ public HudiTableHandle(
this.limit = requireNonNull(limit, "limit is null");
this.hudiTableSchema = requireNonNull(hudiTableSchema, "hudiTableSchema is null");
this.lazyLatestCommitTime = Lazy.lazily(latestCommitTimeSupplier);
+ this.lazyTableConfig = requireNonNull(lazyTableConfig, "lazyTableConfig is null");
+ this.lazyCommittedInstants = requireNonNull(lazyCommittedInstants, "lazyCommittedInstants is null");
+ this.lazyFileGroupReaderTableState = Lazy.lazily(() -> buildFileGroupReaderTableState(basePath, lazyTableConfig.get(), lazyCommittedInstants.get()));
+ }
+
+ /**
+ * The table properties workers need. The create schema is left out: it can be as large as the table schema, and
+ * workers read the schema from {@link #getTableSchemaStr()}.
+ */
+ private static Map<String, String> toMap(HoodieTableConfig tableConfig)
+ {
+ Properties properties = tableConfig.getProps();
+ return properties.stringPropertyNames().stream()
+ .filter(key -> !key.equals(HoodieTableConfig.CREATE_SCHEMA.key()))
+ .collect(toImmutableMap(identity(), properties::getProperty));
+ }
+
+ /**
+ * Only file groups with log files need the committed instants, and only for tables before version 8, whose log
+ * blocks the reader checks against the timeline.
+ */
+ private static Optional<HudiCommittedInstants> captureCommittedInstants(HoodieTableMetaClient metaClient, HoodieTableType tableType, String latestCommitTime)
Review Comment:
Just an FYI, we do not have tests covering v6/v7 mor tables for this part. None of our tests have an inflight or requested commit without a matching completed file.
Oerhaps we could add a Trino trests tusing `PayloadOnlyMergerHudiTablesInitializer` that writes c1, c2, c3 to a v6 table, delete c2's completed .deltacommit file, then check if the _rt returns a base value for c2's keys. This proves that the cutoff at latestCommitTime keeps c2 in the inflight set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(trino): read file groups on workers from table state shipped on the handle [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20086:
URL: https://github.com/apache/hudi/pull/20086#issuecomment-5852660023
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20086?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `93.51852%` with `28 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.42%. Comparing base ([`463b615`](https://app.codecov.io/gh/apache/hudi/commit/463b6159fe7fae8f9132acda3f473233fd265a5d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5f9c388`](https://app.codecov.io/gh/apache/hudi/commit/5f9c3880c9afb7ee1a8254fea671e91780257739?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20086?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ala/org/apache/hudi/cdc/CDCFileGroupIterator.scala](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2Fcdc%2FCDCFileGroupIterator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9jZGMvQ0RDRmlsZUdyb3VwSXRlcmF0b3Iuc2NhbGE=) | 57.89% | [4 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ources/parquet/HoodieFileGroupReaderFunction.scala](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FHoodieFileGroupReaderFunction.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9wYXJxdWV0L0hvb2RpZUZpbGVHcm91cFJlYWRlckZ1bmN0aW9uLnNjYWxh) | 97.64% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/hudi/common/table/log/HoodieLogFormatReader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Flog%2FHoodieLogFormatReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRSZWFkZXIuamF2YQ==) | 70.00% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...mon/table/read/lsm/LsmFileGroupRecordIterator.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Flsm%2FLsmFileGroupRecordIterator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvbHNtL0xzbUZpbGVHcm91cFJlY29yZEl0ZXJhdG9yLmphdmE=) | 57.14% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/hudi/common/schema/internal/InternalSchema.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2Finternal%2FInternalSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9pbnRlcm5hbC9JbnRlcm5hbFNjaGVtYS5qYXZh) | 93.75% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../common/table/log/HoodieMergedLogRecordReader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Flog%2FHoodieMergedLogRecordReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVNZXJnZWRMb2dSZWNvcmRSZWFkZXIuamF2YQ==) | 50.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ommon/table/read/lsm/HoodieLsmFileGroupReader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Flsm%2FHoodieLsmFileGroupReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvbHNtL0hvb2RpZUxzbUZpbGVHcm91cFJlYWRlci5qYXZh) | 84.61% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...parquet/HoodieFileGroupReaderBasedFileFormat.scala](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FHoodieFileGroupReaderBasedFileFormat.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9wYXJxdWV0L0hvb2RpZUZpbGVHcm91cFJlYWRlckJhc2VkRmlsZUZvcm1hdC5zY2FsYQ==) | 90.90% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...i/common/table/read/FileGroupReaderTableState.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2FFileGroupReaderTableState.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvRmlsZUdyb3VwUmVhZGVyVGFibGVTdGF0ZS5qYXZh) | 98.11% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../hudi/common/table/read/HoodieFileGroupReader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2FHoodieFileGroupReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvSG9vZGllRmlsZUdyb3VwUmVhZGVyLmphdmE=) | 92.30% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20086 +/- ##
============================================
+ Coverage 80.40% 80.42% +0.02%
- Complexity 34863 34886 +23
============================================
Files 2546 2549 +3
Lines 142888 143071 +183
Branches 17373 17421 +48
============================================
+ Hits 114891 115067 +176
- Misses 20079 20090 +11
+ Partials 7918 7914 -4
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.99% <93.63%> (+0.03%)` | :arrow_up: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.46% <ø> (+<0.01%)` | :arrow_up: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.74% <ø> (+0.01%)` | :arrow_up: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.91% <93.39%> (+0.06%)` | :arrow_up: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.17% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.05% <ø> (+0.18%)` | :arrow_up: |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.99% <ø> (+0.01%)` | :arrow_up: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.99% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.06% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20086/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (-0.77%)` | :arrow_down: |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20086/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20086/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.98% <46.36%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20086?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../org/apache/hudi/common/schema/internal/Types.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2Finternal%2FTypes.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9pbnRlcm5hbC9UeXBlcy5qYXZh) | `83.06% <100.00%> (+0.33%)` | :arrow_up: |
| [...di/common/table/log/BaseHoodieLogRecordReader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Flog%2FBaseHoodieLogRecordReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9CYXNlSG9vZGllTG9nUmVjb3JkUmVhZGVyLmphdmE=) | `85.28% <100.00%> (+0.30%)` | :arrow_up: |
| [...ommon/table/log/HoodieLogBlockMetadataScanner.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Flog%2FHoodieLogBlockMetadataScanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dCbG9ja01ldGFkYXRhU2Nhbm5lci5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...ache/hudi/common/table/read/CommittedInstants.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2FCommittedInstants.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvQ29tbWl0dGVkSW5zdGFudHMuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ommon/table/read/FileGroupReaderSchemaHandler.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2FFileGroupReaderSchemaHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvRmlsZUdyb3VwUmVhZGVyU2NoZW1hSGFuZGxlci5qYXZh) | `98.18% <100.00%> (+0.03%)` | :arrow_up: |
| [.../table/read/ParquetRowIndexBasedSchemaHandler.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2FParquetRowIndexBasedSchemaHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvUGFycXVldFJvd0luZGV4QmFzZWRTY2hlbWFIYW5kbGVyLmphdmE=) | `92.85% <100.00%> (+0.54%)` | :arrow_up: |
| [...ead/buffer/DefaultFileGroupRecordBufferLoader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Fbuffer%2FDefaultFileGroupRecordBufferLoader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvYnVmZmVyL0RlZmF1bHRGaWxlR3JvdXBSZWNvcmRCdWZmZXJMb2FkZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ommon/table/read/buffer/FileGroupRecordBuffer.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Fbuffer%2FFileGroupRecordBuffer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvYnVmZmVyL0ZpbGVHcm91cFJlY29yZEJ1ZmZlci5qYXZh) | `95.18% <ø> (ø)` | |
| [...table/read/buffer/FileGroupRecordBufferLoader.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Fbuffer%2FFileGroupRecordBufferLoader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvYnVmZmVyL0ZpbGVHcm91cFJlY29yZEJ1ZmZlckxvYWRlci5qYXZh) | `100.00% <ø> (ø)` | |
| [...ble/read/buffer/KeyBasedFileGroupRecordBuffer.java](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fread%2Fbuffer%2FKeyBasedFileGroupRecordBuffer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3JlYWQvYnVmZmVyL0tleUJhc2VkRmlsZUdyb3VwUmVjb3JkQnVmZmVyLmphdmE=) | `92.30% <100.00%> (+0.41%)` | :arrow_up: |
| ... and [20 more](https://app.codecov.io/gh/apache/hudi/pull/20086?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [21 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20086/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org