You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/07/06 21:04:52 UTC
[PR] feat(cli): add table set-meta-fields-mode command [hudi]
nsivabalan opened a new pull request, #19206:
URL: https://github.com/apache/hudi/pull/19206
### Describe the issue this Pull Request addresses
Adds a hudi-cli command to change `hoodie.meta.fields.mode` on an existing table. This is the sanctioned way to change the property outside of table creation — the property is immutable at runtime because it is a physical-storage decision baked into files at write time.
**Stacked on top of #19205.** This PR is best reviewed after #19205 lands — the base of the diff currently includes PR-A's changes. Once #19205 merges to master, this PR will show only the ~209-line CLI addition.
### Summary and Changelog
New shell command:
```
table set-meta-fields-mode --target-mode <MODE> [--force true|false]
```
Where `<MODE>` is one of `ALL`, `NONE`, `COMMIT_TIME_ONLY`, `FILE_NAME_ONLY`, `COMMIT_TIME_AND_FILE_NAME`.
#### Safety guard
On a table that already has commits, the command refuses to change the mode by default:
```
Refusing to change hoodie.meta.fields.mode on a table that already has N commit(s).
Existing files were written under X and will not be rewritten by this command; new
commits would be written under Y, producing mixed-mode files whose incremental /
file-pruning semantics differ between old and new data. Pass --force if you accept
the consequences, or recreate the table to change the mode cleanly.
```
Pass `--force` to override. A warning is logged describing the data-correctness impact.
#### Behavior
- **ALL / NONE**: persisted implicitly via `populate.meta.fields=true/false`; the `hoodie.meta.fields.mode` property is *cleared* when transitioning to ALL/NONE.
- **COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME**: persisted explicitly on `hoodie.properties` alongside `populate.meta.fields=false`.
- **No-op** when the target matches the current mode.
- **Unknown enum value**: rejected up-front with a clear error.
#### Test coverage (\`TestTableCommand\`)
- All 5 modes settable on a fresh table.
- Selective → ALL clears the property.
- No-op when the target matches the current mode.
- Rejects unknown enum values.
- Refuses on a populated table without \`--force\`; mode does not change.
- Accepts on a populated table with \`--force\`; mode changes.
All 9 tests pass locally.
### Impact
- **Storage layout**: no change unless the user invokes the command.
- **API**: additive — new CLI command, no signature changes to existing commands.
- **Configuration**: no new properties.
- **Performance**: N/A.
- **Forward-compat**: N/A.
### Risk Level
low
Narrow, additive change with a strong default-safe guard. \`--force\` is required to change mode on a populated table.
### Documentation Update
- New command documented via the \`@ShellMethod\` value string.
- If the CLI docs page exists, a separate docs PR will add an entry.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3769194704
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/TenToNineDowngradeHandler.java:
##########
@@ -18,25 +18,144 @@
package org.apache.hudi.table.upgrade;
+import org.apache.hudi.common.config.ConfigProperty;
import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.MetaFieldsMode;
import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
-import java.util.Collections;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
/**
* Version 10 writes native log files by default. Downgrading to version 9 requires
* full compaction of native data/delete logs before the downgrade completes.
+ *
+ * <p>Version 10 also introduced {@code hoodie.meta.fields.mode}. {@code hoodie.populate.meta.fields}
+ * is always written back from it ({@code ALL -> true}, every other mode {@code -> false}), mirroring
+ * how {@link NineToEightDowngradeHandler} restores {@code hoodie.table.payload.class}. That restate
+ * is load-bearing: {@code POPULATE_META_FIELDS} defaults to {@code true}, so a table carrying only
+ * the mode would otherwise downgrade to {@code ALL} and claim {@code _hoodie_record_key} is
+ * populated on files where it is null.
+ *
+ * <p>What happens to the mode itself depends on whether the legacy boolean can express it:
+ *
+ * <ul>
+ * <li>{@link MetaFieldsMode#ALL} / {@link MetaFieldsMode#NONE} — dropped. These are exactly the
+ * two states the boolean expresses, so the mode carries nothing the downgraded table lacks.</li>
+ * <li>A selective mode, restated by the writer — <b>retained</b>. Restating it is the operator
+ * asserting that every reader of this table honors the mode rather than the boolean alone.
+ * Keeping it is also what makes the round trip lossless: a later re-upgrade finds the mode
+ * intact rather than deriving {@code NONE} from the boolean.</li>
+ * <li>A selective mode, not restated (or restated as a different value) — <b>rejected</b>.
+ * Dropping it would collapse the table to {@code NONE} irreversibly, and that is not a call to
+ * make on the operator's behalf.</li>
+ * </ul>
+ *
+ * <p>Retaining the mode on a version 9 table is safe mechanically: the property carries no
+ * {@code sinceVersion}, so {@code dropInvalidConfigs} does not strip it on load.
*/
public class TenToNineDowngradeHandler implements DowngradeHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TenToNineDowngradeHandler.class);
+
+ /**
+ * The meta-fields mode the writer explicitly asked for, if any.
+ *
+ * <p>Empty when the property is absent or blank -- which is the ordinary case, since almost no
+ * caller restates meta-field settings. Restating it during a downgrade is therefore a deliberate
+ * signal rather than something that happens by accident, which is what makes it usable as consent.
+ */
+ private static Option<MetaFieldsMode> statedMetaFieldsMode(HoodieWriteConfig config) {
+ if (config == null || !config.contains(HoodieTableConfig.META_FIELDS_MODE)) {
+ return Option.empty();
+ }
+ String raw = config.getString(HoodieTableConfig.META_FIELDS_MODE);
+ return StringUtils.isNullOrEmpty(raw) ? Option.empty() : Option.of(MetaFieldsMode.parse(raw));
+ }
+
@Override
public UpgradeDowngrade.TableConfigChangeSet downgrade(
HoodieWriteConfig config,
HoodieEngineContext context,
String instantTime,
SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ Set<ConfigProperty> propertiesToDelete = new HashSet<>();
+ propertiesToDelete.add(HoodieTableConfig.TABLE_STORAGE_LAYOUT);
+
+ Map<ConfigProperty, String> propertiesToUpdate = new HashMap<>();
+ if (upgradeDowngradeHelper != null) {
+ MetaFieldsMode metaFieldsMode =
+ upgradeDowngradeHelper.getTable(config, context).getMetaClient().getTableConfig().getMetaFieldsMode();
+
+ // Always restate the legacy boolean from the mode. Version 9 readers understand only that
+ // property, and without it POPULATE_META_FIELDS falls back to its `true` default -- i.e. the
+ // table silently downgrades to ALL and claims meta columns it does not have. For ALL / NONE
+ // this restates what was already there; for a selective mode it writes `false`, so a reader
+ // that does not honor the mode under-claims rather than over-claims.
+ propertiesToUpdate.put(HoodieTableConfig.POPULATE_META_FIELDS,
+ String.valueOf(metaFieldsMode.toLegacyPopulateMetaFields()));
+
+ if (!metaFieldsMode.isSelective()) {
+ // ALL and NONE are exactly what the boolean can express, so the mode carries no information
+ // the downgraded table lacks. Drop it.
+ propertiesToDelete.add(HoodieTableConfig.META_FIELDS_MODE);
+ } else if (statedMetaFieldsMode(config).map(metaFieldsMode::equals).orElse(false)) {
+ // Selective mode, and the writer restated it. That restatement is the operator asserting
+ // "I know this table is selective and every reader of it honors the mode" -- so the mode is
+ // retained on the downgraded table rather than dropped. Two consequences they are taking on:
+ // the property outlives the version that formally understands it (harmless -- it carries no
+ // sinceVersion, so it is not stripped on load), and a reader that ignores it sees
+ // populate.meta.fields=false and treats the table as NONE.
+ //
+ // Keeping it is also what makes the round trip lossless: a later re-upgrade finds the mode
+ // intact instead of deriving NONE from the boolean.
+ LOG.warn("Downgrading a table on {}={} to table version 9. The mode is being retained on the "
+ + "downgraded table because it was explicitly restated on the writer. Version 9 does "
+ + "not formally understand it, so every reader of this table must honor {} rather "
+ + "than relying on {} alone -- which now reads false.",
+ HoodieTableConfig.META_FIELDS_MODE.key(), metaFieldsMode,
+ HoodieTableConfig.META_FIELDS_MODE.key(),
+ HoodieTableConfig.POPULATE_META_FIELDS.key());
+ } else {
+ // Selective mode and the writer did not restate it (or restated a different one). Dropping
+ // the mode here would silently collapse the table to NONE and could not be undone: a
+ // re-upgrade derives the mode from the boolean, which by then reads false. Rather than make
+ // that call on the operator's behalf, require them to say so explicitly.
+ throw new HoodieUpgradeDowngradeException(String.format(
+ "Cannot downgrade to table version 9: %s=%s populates only some meta columns, which "
+ + "version 9 cannot express. Dropping it would collapse the table to NONE and could "
+ + "not be reversed by upgrading again. Set %s=%s on the writer to confirm you want "
+ + "the mode retained on the downgraded table -- every reader must then honor it "
Review Comment:
🤖 nit: the comment block "No helper means the table config is unreachable..." appears twice -- once inside the `else` block and again right after its closing brace. The second occurrence looks like a copy-paste artifact; could you remove it?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/common/model/MetaFieldsMode.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.Locale;
+
+/**
+ * Which of Hudi's meta columns are physically populated on disk.
+ *
+ * <p>Selective modes exist so that tables that opt out of the default {@code populate.meta.fields=true}
+ * can still keep the two columns that matter for downstream operations without paying for the other
+ * three:
+ *
+ * <ul>
+ * <li>{@code _hoodie_commit_time} — required for incremental queries.</li>
+ * <li>{@code _hoodie_file_name} — useful for file-level pruning / investigation lookups.</li>
+ * </ul>
+ *
+ * <p>The remaining three meta columns ({@code _hoodie_commit_seqno}, {@code _hoodie_record_key},
+ * {@code _hoodie_partition_path}) are all-or-nothing — either populate every meta column ({@link #ALL})
+ * or none of them beyond the two selectable ones. If you need any of the remaining columns, set
+ * {@code hoodie.populate.meta.fields=true}.
+ *
+ * <p>This enum is the single source of truth for meta-column population. The legacy boolean
+ * {@code hoodie.populate.meta.fields} is deprecated and consulted only when
+ * {@code hoodie.meta.fields.mode} is absent, so that tables written before the mode property
+ * existed keep their behavior:
+ *
+ * <ul>
+ * <li>{@code populate.meta.fields=true} (or absent) → {@link #ALL} — today's default.</li>
+ * <li>{@code populate.meta.fields=false} → {@link #NONE}.</li>
+ * </ul>
+ *
+ * <p>On-disk representation: the enum {@link #name()} is persisted in {@code hoodie.properties}
+ * under the property {@code hoodie.meta.fields.mode}.
+ */
+public enum MetaFieldsMode {
+ /**
+ * All five Hudi meta columns are populated — today's default.
+ */
+ ALL(true, true),
+
+ /**
+ * No Hudi meta columns are populated. Incremental queries are unsupported. File-level pruning
+ * that depends on {@code _hoodie_file_name} is unsupported.
+ */
+ NONE(false, false),
+
+ /**
+ * Only {@code _hoodie_commit_time} is populated. Incremental queries remain functional; other
+ * meta columns stay null on disk.
+ */
+ COMMIT_TIME_ONLY(true, false),
+
+ /**
+ * Only {@code _hoodie_file_name} is populated. Useful for file-level lookups and debugging;
+ * incremental queries are unsupported.
+ */
+ FILE_NAME_ONLY(false, true),
+
+ /**
+ * Both {@code _hoodie_commit_time} and {@code _hoodie_file_name} are populated.
+ */
+ COMMIT_TIME_AND_FILE_NAME(true, true);
+
+ private final boolean commitTimePopulated;
+ private final boolean fileNamePopulated;
+
+ MetaFieldsMode(boolean commitTimePopulated, boolean fileNamePopulated) {
+ this.commitTimePopulated = commitTimePopulated;
+ this.fileNamePopulated = fileNamePopulated;
+ }
+
+ public boolean isCommitTimePopulated() {
+ return commitTimePopulated;
+ }
+
+ public boolean isFileNamePopulated() {
+ return fileNamePopulated;
+ }
+
+ /**
+ * @return true when all five meta columns are populated (i.e. this is {@link #ALL}). Selective
+ * modes never populate {@code _hoodie_record_key}, {@code _hoodie_partition_path}, or
+ * {@code _hoodie_commit_seqno}.
+ */
+ public boolean isRecordKeyPopulated() {
+ return this == ALL;
+ }
+
+ /**
+ * @return true for the modes that populate some but not all meta columns, i.e. everything except
+ * {@link #ALL} and {@link #NONE}.
+ *
+ * <p>These are the modes the deprecated {@code hoodie.populate.meta.fields} boolean cannot
+ * express, so they are what callers gate on when a code path only understands all-or-nothing meta
+ * fields — writer engines not yet wired for selective population, table versions that predate the
+ * mode property, and validation that must not let a two-state writer speak for a five-state table.
+ */
+ public boolean isSelective() {
+ return this != ALL && this != NONE;
+ }
+
+ /**
+ * Resolve the effective mode from any {@link HoodieConfig} that may carry the two properties —
+ * a table config, a write config, or a bare config built from write options. Preferred over the
+ * two-argument overload: it keeps the property keys and the precedence rule in one place instead
+ * of repeating them at every call site.
+ */
+ public static MetaFieldsMode resolve(HoodieConfig config) {
+ return resolve(config.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE),
+ config.getBooleanOrDefault(HoodieTableConfig.POPULATE_META_FIELDS));
+ }
+
+ /**
+ * Resolve the effective mode. {@code hoodie.meta.fields.mode} is the source of truth; the
+ * deprecated {@code hoodie.populate.meta.fields} boolean is a fallback for tables written before
+ * the mode property existed. Precedence:
+ *
+ * <ul>
+ * <li>non-empty mode → the parsed enum value (the legacy boolean is not consulted).</li>
+ * <li>null/empty mode + {@code populateMetaFields=false} → {@link #NONE}.</li>
+ * <li>null/empty mode + {@code populateMetaFields=true} → {@link #ALL}.</li>
+ * </ul>
+ *
+ * @param rawMode raw {@code hoodie.meta.fields.mode} value; may be null or empty.
+ * @param legacyPopulateMetaFields value of the deprecated {@code hoodie.populate.meta.fields}.
+ * @throws IllegalArgumentException when the raw mode value does not match any enum value. This
+ * includes the pre-enum comma-separated format — callers that upgrade an old table must
+ * migrate the value through the hudi-cli.
+ */
+ public static MetaFieldsMode resolve(String rawMode, boolean legacyPopulateMetaFields) {
+ if (StringUtils.isNullOrEmpty(rawMode)) {
+ return legacyPopulateMetaFields ? ALL : NONE;
+ }
+ return parse(rawMode);
+ }
+
+ /**
+ * Resolve the mode from a table config that may not be able to answer, defaulting to {@link #ALL}.
+ *
+ * <p>For write handles, which read the mode from the table rather than the write config. A real
+ * {@link org.apache.hudi.common.table.HoodieTableConfig#getMetaFieldsMode()} never returns null --
+ * it falls back through the deprecated boolean to {@code ALL}. But a handle constructed against a
+ * partially-stubbed table (as several unit tests do) would otherwise take a null here and NPE later,
+ * at the point of use, far from the cause. {@code ALL} is the safe default: it is the pre-feature
+ * behavior, so a caller that cannot state a mode gets what it would have got before this existed.
+ */
+ public static MetaFieldsMode orAllIfUnknown(MetaFieldsMode mode) {
+ return mode == null ? ALL : mode;
+ }
+
+ /**
+ * Parse a raw {@code hoodie.meta.fields.mode} value into an enum constant, with a message that
+ * lists the allowed values. Prefer this over {@link #valueOf(String)} for user-supplied input.
+ */
+ public static MetaFieldsMode parse(String rawMode) {
+ try {
+ // Case-insensitive: users hand-editing hoodie.properties or passing write options should not
+ // have to match the enum's casing exactly.
+ return MetaFieldsMode.valueOf(rawMode.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String.format(
+ "Unsupported value '%s' for hoodie.meta.fields.mode. Allowed values: %s, %s, %s, %s, %s.",
+ rawMode, ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME), e);
Review Comment:
🤖 nit: the error message hardcodes all five enum values by name -- adding a sixth mode would require updating this line manually. Something like `Arrays.stream(values()).map(MetaFieldsMode::name).collect(Collectors.joining(", "))` would stay correct automatically.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieStreamerMetaFieldsMode.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.MetaFieldsMode;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.functions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * End-to-end coverage for {@code hoodie.meta.fields.mode} through the HoodieStreamer entrypoint.
+ * Each parameterized invocation runs a single ingest cycle in the given {@link MetaFieldsMode} and
+ * verifies both the persisted table property and the actual on-disk parquet column population.
+ *
+ * <p>Rejection paths (unknown token, populate=true+mode, MoR+mode) are exercised in the datasource
+ * test {@code TestMetaFieldsMode}; this fixture focuses on the streamer control-flow.
+ */
+public class TestHoodieStreamerMetaFieldsMode extends HoodieDeltaStreamerTestBase {
+
+ /**
+ * Only the selective modes are parameterized here. ALL and NONE add no mode key at all, so they
+ * exercise none of the streamer-side plumbing this feature introduced — they are covered by the
+ * datasource tests and by {@code TestHoodieTableConfig}'s resolution cases.
+ */
+ @ParameterizedTest
+ @EnumSource(value = MetaFieldsMode.class,
+ names = {"COMMIT_TIME_ONLY", "FILE_NAME_ONLY", "COMMIT_TIME_AND_FILE_NAME"})
+ public void testStreamerRespectsMetaFieldsMode(MetaFieldsMode mode) throws Exception {
+ String tablePath = basePath + "/streamer_meta_fields_mode_" + mode.name();
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ // Force CoW; selective modes are CoW-only until MoR log-write is wired.
+ cfg.tableType = "COPY_ON_WRITE";
+ // The mode alone — pairing it with populate.meta.fields would now be a stated conflict, since the
+ // mode is authoritative and the boolean is only the fallback for resolving an absent one.
+ cfg.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + mode.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(cfg, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(mode, metaClient.getTableConfig().getMetaFieldsMode(),
+ "streamer must persist mode=" + mode + " on hoodie.properties");
+ assertOnDiskMetaColumns(tablePath, mode);
+ }
+
+ /**
+ * The regression this fixture exists for, and the one cshuo raised in review: a restarted streamer
+ * that does not restate the mode must not silently write the wrong meta columns.
+ *
+ * <p>StreamSync builds its write config from {@code props} alone, and the mode is persisted only by
+ * {@code initializeEmptyTable}, which runs solely when the base path does not exist. So a second run
+ * used to resolve to {@link MetaFieldsMode#NONE} and write base files with a null
+ * {@code _hoodie_commit_time} while {@code hoodie.properties} still advertised
+ * {@code COMMIT_TIME_ONLY} — incremental queries were then admitted and silently dropped every one
+ * of those rows.
+ *
+ * <p>A selective table now requires the writer to state the mode, so the restart is <b>rejected</b>
+ * rather than inheriting. Inheritance would work here, but it cannot be relied on everywhere: the
+ * write path reads the mode in factories and handles that hold no table config at all, so the write
+ * config has to be right on its own. Failing loudly at init is what makes that guarantee real —
+ * and a rejected run leaves the table exactly as it was, which silent narrowing did not.
+ * See {@code testRestartRestatingTheModeSucceeds} for the migration path.
+ */
+ @Test
+ public void testRestartWithoutRestatingTheModeIsRejected() throws Exception {
+ String tablePath = basePath + "/streamer_restart_inherits_mode";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieTestUtils.createMetaClient(context, tablePath).getTableConfig().getMetaFieldsMode());
+
+ long commitsAfterFirstRun = HoodieTestUtils.createMetaClient(context, tablePath)
+ .getActiveTimeline().filterCompletedInstants().countInstants();
+
+ // Restart against the existing table stating neither the mode nor the legacy boolean.
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected the writer to be told to state the mode, got: " + rootMessage);
+
+ // The rejected run must have left the table untouched -- no mode change, no extra commit, and no
+ // base file carrying a null commit time. That last one is the actual data loss being prevented:
+ // such rows are admitted by incremental queries and then silently dropped.
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "a rejected restart must leave the table's mode untouched");
+ assertEquals(commitsAfterFirstRun,
+ metaClient.getActiveTimeline().filterCompletedInstants().countInstants(),
+ "a rejected restart must not have committed anything");
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0,
+ raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time on a COMMIT_TIME_ONLY table");
+ }
+
+ /**
+ * The migration path for the case above: a restart that restates the table's mode proceeds normally.
+ *
+ * <p>This is what every writer against a selective table must do, and it is the assertion that keeps
+ * the requirement from being a dead end -- the mode is stateable, and stating it is enough.
+ */
+ @Test
+ public void testRestartRestatingTheModeSucceeds() throws Exception {
+ String tablePath = basePath + "/streamer_restart_restates_mode";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ // Restart restating the same mode.
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+ restart.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "the restart must not have changed the table's mode");
+ assertTrue(metaClient.getActiveTimeline().filterCompletedInstants().countInstants() >= 2,
+ "expected the restart to have produced a second commit");
+
+ // Every row across both commits carries a commit time. A row with a null one is what incremental
+ // queries silently drop, so this is the assertion that catches the regression.
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0,
+ raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time on a COMMIT_TIME_ONLY table");
+ assertTrue(raw.select(HoodieRecord.COMMIT_TIME_METADATA_FIELD).distinct().count() >= 2,
+ "both commits must be represented, so the second run really did write through this path");
+ }
+
+ /**
+ * The variant @voonhous asked for, and the one cshuo originally described: the restart states the
+ * deprecated boolean rather than nothing at all.
+ *
+ * <p>Both cases are rejected against a selective table, for the same reason: the writer has not
+ * stated the mode the table is on. Stating the boolean instead makes the disagreement explicit
+ * rather than merely unstated, but the outcome is the same. Before this rule it narrowed silently,
+ * writing base files with a null {@code _hoodie_commit_time} into a table that still advertised the
+ * mode.
+ */
+ @Test
+ public void testRestartStatingTheLegacyBooleanIsRejected() throws Exception {
+ String tablePath = basePath + "/streamer_restart_legacy_boolean_conflict";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
Review Comment:
🤖 nit: could you scrub the reviewer names ("@voonhous", "cshuo") from the Javadoc before merging? That context makes sense in a PR thread but not in committed source — future readers have no way to look them up.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieStreamerMetaFieldsMode.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.MetaFieldsMode;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.functions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * End-to-end coverage for {@code hoodie.meta.fields.mode} through the HoodieStreamer entrypoint.
+ * Each parameterized invocation runs a single ingest cycle in the given {@link MetaFieldsMode} and
+ * verifies both the persisted table property and the actual on-disk parquet column population.
+ *
+ * <p>Rejection paths (unknown token, populate=true+mode, MoR+mode) are exercised in the datasource
+ * test {@code TestMetaFieldsMode}; this fixture focuses on the streamer control-flow.
+ */
+public class TestHoodieStreamerMetaFieldsMode extends HoodieDeltaStreamerTestBase {
+
+ /**
+ * Only the selective modes are parameterized here. ALL and NONE add no mode key at all, so they
+ * exercise none of the streamer-side plumbing this feature introduced — they are covered by the
+ * datasource tests and by {@code TestHoodieTableConfig}'s resolution cases.
+ */
+ @ParameterizedTest
+ @EnumSource(value = MetaFieldsMode.class,
+ names = {"COMMIT_TIME_ONLY", "FILE_NAME_ONLY", "COMMIT_TIME_AND_FILE_NAME"})
+ public void testStreamerRespectsMetaFieldsMode(MetaFieldsMode mode) throws Exception {
+ String tablePath = basePath + "/streamer_meta_fields_mode_" + mode.name();
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ // Force CoW; selective modes are CoW-only until MoR log-write is wired.
+ cfg.tableType = "COPY_ON_WRITE";
+ // The mode alone — pairing it with populate.meta.fields would now be a stated conflict, since the
+ // mode is authoritative and the boolean is only the fallback for resolving an absent one.
+ cfg.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + mode.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(cfg, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(mode, metaClient.getTableConfig().getMetaFieldsMode(),
+ "streamer must persist mode=" + mode + " on hoodie.properties");
+ assertOnDiskMetaColumns(tablePath, mode);
+ }
+
+ /**
+ * The regression this fixture exists for, and the one cshuo raised in review: a restarted streamer
+ * that does not restate the mode must not silently write the wrong meta columns.
+ *
+ * <p>StreamSync builds its write config from {@code props} alone, and the mode is persisted only by
+ * {@code initializeEmptyTable}, which runs solely when the base path does not exist. So a second run
+ * used to resolve to {@link MetaFieldsMode#NONE} and write base files with a null
+ * {@code _hoodie_commit_time} while {@code hoodie.properties} still advertised
+ * {@code COMMIT_TIME_ONLY} — incremental queries were then admitted and silently dropped every one
+ * of those rows.
+ *
+ * <p>A selective table now requires the writer to state the mode, so the restart is <b>rejected</b>
+ * rather than inheriting. Inheritance would work here, but it cannot be relied on everywhere: the
+ * write path reads the mode in factories and handles that hold no table config at all, so the write
+ * config has to be right on its own. Failing loudly at init is what makes that guarantee real —
+ * and a rejected run leaves the table exactly as it was, which silent narrowing did not.
+ * See {@code testRestartRestatingTheModeSucceeds} for the migration path.
+ */
+ @Test
+ public void testRestartWithoutRestatingTheModeIsRejected() throws Exception {
+ String tablePath = basePath + "/streamer_restart_inherits_mode";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieTestUtils.createMetaClient(context, tablePath).getTableConfig().getMetaFieldsMode());
+
+ long commitsAfterFirstRun = HoodieTestUtils.createMetaClient(context, tablePath)
+ .getActiveTimeline().filterCompletedInstants().countInstants();
+
+ // Restart against the existing table stating neither the mode nor the legacy boolean.
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected the writer to be told to state the mode, got: " + rootMessage);
+
+ // The rejected run must have left the table untouched -- no mode change, no extra commit, and no
+ // base file carrying a null commit time. That last one is the actual data loss being prevented:
+ // such rows are admitted by incremental queries and then silently dropped.
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "a rejected restart must leave the table's mode untouched");
+ assertEquals(commitsAfterFirstRun,
+ metaClient.getActiveTimeline().filterCompletedInstants().countInstants(),
+ "a rejected restart must not have committed anything");
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0,
+ raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time on a COMMIT_TIME_ONLY table");
+ }
+
+ /**
+ * The migration path for the case above: a restart that restates the table's mode proceeds normally.
+ *
+ * <p>This is what every writer against a selective table must do, and it is the assertion that keeps
+ * the requirement from being a dead end -- the mode is stateable, and stating it is enough.
+ */
+ @Test
+ public void testRestartRestatingTheModeSucceeds() throws Exception {
+ String tablePath = basePath + "/streamer_restart_restates_mode";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ // Restart restating the same mode.
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+ restart.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "the restart must not have changed the table's mode");
+ assertTrue(metaClient.getActiveTimeline().filterCompletedInstants().countInstants() >= 2,
+ "expected the restart to have produced a second commit");
+
+ // Every row across both commits carries a commit time. A row with a null one is what incremental
+ // queries silently drop, so this is the assertion that catches the regression.
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0,
+ raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time on a COMMIT_TIME_ONLY table");
+ assertTrue(raw.select(HoodieRecord.COMMIT_TIME_METADATA_FIELD).distinct().count() >= 2,
+ "both commits must be represented, so the second run really did write through this path");
+ }
+
+ /**
+ * The variant @voonhous asked for, and the one cshuo originally described: the restart states the
+ * deprecated boolean rather than nothing at all.
+ *
+ * <p>Both cases are rejected against a selective table, for the same reason: the writer has not
+ * stated the mode the table is on. Stating the boolean instead makes the disagreement explicit
+ * rather than merely unstated, but the outcome is the same. Before this rule it narrowed silently,
+ * writing base files with a null {@code _hoodie_commit_time} into a table that still advertised the
+ * mode.
+ */
+ @Test
+ public void testRestartStatingTheLegacyBooleanIsRejected() throws Exception {
+ String tablePath = basePath + "/streamer_restart_legacy_boolean_conflict";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+ restart.configs.add(HoodieTableConfig.POPULATE_META_FIELDS.key() + "=false");
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key())
+ || rootMessage.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected a meta-fields conflict, got: " + rootMessage);
+
+ // The failed run must not have changed the table, nor written rows with a null commit time.
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "a rejected restart must leave the table's mode untouched");
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0, raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time");
+ }
+
+ @Test
+ public void testStreamerRejectsMorWithSelectiveMode() throws Exception {
+ String tablePath = basePath + "/streamer_mor_selective_rejected";
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tablePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = "MERGE_ON_READ";
+ cfg.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(cfg, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains("COPY_ON_WRITE") || rootMessage.contains("MERGE_ON_READ")
+ || rootMessage.contains("MoR") || rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "Expected MoR-restriction error, got: " + rootMessage);
+ }
+
+ private void assertOnDiskMetaColumns(String tablePath, MetaFieldsMode expectedMode) {
+ // Default HoodieTestDataGenerator partitions are YYYY/MM/DD (three levels).
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ Row first = raw.select(
+ HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+ HoodieRecord.COMMIT_SEQNO_METADATA_FIELD,
+ HoodieRecord.RECORD_KEY_METADATA_FIELD,
+ HoodieRecord.PARTITION_PATH_METADATA_FIELD,
+ HoodieRecord.FILENAME_METADATA_FIELD).first();
+
+ if (expectedMode.isCommitTimePopulated()) {
+ assertNotNull(first.get(0), "commit_time must be populated in mode " + expectedMode);
+ } else {
+ assertNull(first.get(0), "commit_time must be null in mode " + expectedMode);
+ }
+ if (expectedMode.isFileNamePopulated()) {
+ assertNotNull(first.get(4), "file_name must be populated in mode " + expectedMode);
+ } else {
+ assertNull(first.get(4), "file_name must be null in mode " + expectedMode);
+ }
+ if (expectedMode == MetaFieldsMode.ALL) {
+ assertNotNull(first.get(1), "commit_seq_no must be populated in ALL mode");
+ assertNotNull(first.get(2), "record_key must be populated in ALL mode");
+ assertNotNull(first.get(3), "partition_path must be populated in ALL mode");
+ } else {
+ assertNull(first.get(1), "commit_seq_no must be null outside ALL mode");
+ assertNull(first.get(2), "record_key must be null outside ALL mode");
+ assertNull(first.get(3), "partition_path must be null outside ALL mode");
+ }
+ }
+
+ private static String rootMessageOf(Throwable thrown) {
+ Throwable root = thrown;
Review Comment:
🤖 nit: `rootMessageOf` is copy-pasted verbatim from `TestMetaFieldsModeE2E` — have you considered pulling it into a shared test helper so there's only one copy to keep in sync?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222088855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed5b5263fa2b6a60abb79ccd696f892230130a7f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188)
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 109193946a3e70e3b589e252b44ad2eb68092699 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5223283332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 109193946a3e70e3b589e252b44ad2eb68092699 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191)
* ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5016573515
Rebased onto the refreshed PR #19205 head (which is itself now rebased on latest master, 2026-07-19). No new code changes on top of what's in #19205 aside from the CLI commit itself.
**Self-review polish:**
- Strengthened `testSetMetaFieldsModeOnFreshTableToCommitTimeOnly` to also assert that the rendered CLI output surfaces the changed properties (`populate.meta.fields` and `meta.fields.mode`), not just the persisted table config. Everything else already had targeted assertions.
**Prior CI failures inherited from PR #19205 (co-author trailer, TestHoodieTableConfig, TestHoodieSparkSqlWriter, TestHoodieStreamerMetaFieldsMode) should now be resolved automatically since we're on the rebased #19205 head.**
**One flaky non-related CI failure from the last run:** `test-flink-1 → ITTestHoodieDataSource.testBatchReadWithLimit` failed with `RuntimeException: Failed to fetch next result` (Flink `CollectResultIterator` flake). Neither this PR nor its predecessor touches Flink code paths — flagging so we can dismiss if it recurs.
**Discussion points for reviewer:**
1. **MoR test for the CLI safety check** — `testSetMetaFieldsModeRefusesOnPopulatedTable` seeds a synthetic `.commit` file (CoW). Since `getCommitsTimeline().countInstants()` covers both commits and deltacommits (per HoodieTimeline javadoc), the guard should already work for MoR — but there's no explicit MoR test. Given MoR support lands in PR-C, is it worth adding a MoR-flavored CLI test here or defer to that PR?
2. **`--force` flag test coverage** — current test passes `--force true` as an explicit string. Spring Shell's `defaultValue = "false"` should handle bare `--force` too, but that's untested. Add a bare-flag test, or leave it since the Spring plumbing is well-covered elsewhere?
Not blocking either way — happy to fold in whichever direction you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5219710776
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5270593756
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19206?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `79.10959%` with `61 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 77.52%. Comparing base ([`b2174b6`](https://app.codecov.io/gh/apache/hudi/commit/b2174b6f9ea30235cabf39d8269f179e9f670a87?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`097c8d9`](https://app.codecov.io/gh/apache/hudi/commit/097c8d95b3abfc325314c7cc053e113ec014f725?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19206?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...n/java/org/apache/hudi/cli/commands/SparkMain.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-cli%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcli%2Fcommands%2FSparkMain.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NwYXJrTWFpbi5qYXZh) | 0.00% | [15 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/hudi/io/storage/HoodieSparkParquetWriter.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2FHoodieSparkParquetWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTcGFya1BhcnF1ZXRXcml0ZXIuamF2YQ==) | 53.33% | [3 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/common/model/HoodieSparkRecord.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fmodel%2FHoodieSparkRecord.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZVNwYXJrUmVjb3JkLmphdmE=) | 37.50% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/common/model/HoodieAvroIndexedRecord.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fmodel%2FHoodieAvroIndexedRecord.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUF2cm9JbmRleGVkUmVjb3JkLmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/hudi/common/table/HoodieTableMetaClient.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2FHoodieTableMetaClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | 82.60% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/hudi/io/storage/row/HoodieRowCreateHandle.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2Frow%2FHoodieRowCreateHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9yb3cvSG9vZGllUm93Q3JlYXRlSGFuZGxlLmphdmE=) | 85.71% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../scala/org/apache/hudi/IncrementalRelationV1.scala](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FIncrementalRelationV1.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9JbmNyZW1lbnRhbFJlbGF0aW9uVjEuc2NhbGE=) | 0.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../scala/org/apache/hudi/IncrementalRelationV2.scala](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FIncrementalRelationV2.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9JbmNyZW1lbnRhbFJlbGF0aW9uVjIuc2NhbGE=) | 0.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/hudi/MergeOnReadIncrementalRelationV1.scala](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FMergeOnReadIncrementalRelationV1.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9NZXJnZU9uUmVhZEluY3JlbWVudGFsUmVsYXRpb25WMS5zY2FsYQ==) | 0.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/hudi/utilities/HoodieDropPartitionsTool.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2FHoodieDropPartitionsTool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZURyb3BQYXJ0aXRpb25zVG9vbC5qYXZh) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [7 more](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19206 +/- ##
============================================
+ Coverage 77.50% 77.52% +0.02%
- Complexity 32862 32921 +59
============================================
Files 2524 2525 +1
Lines 139355 139574 +219
Branches 16747 16783 +36
============================================
+ Hits 108002 108200 +198
- Misses 23781 23795 +14
- Partials 7572 7579 +7
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.26% <87.14%> (-0.02%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.73% <85.91%> (+<0.01%)` | :arrow_up: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.80% <ø> (+0.05%)` | :arrow_up: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.61% <66.66%> (+0.05%)` | :arrow_up: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.63% <42.85%> (-0.02%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.27% <0.00%> (+<0.01%)` | :arrow_up: |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.98% <95.45%> (+0.03%)` | :arrow_up: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.55% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.47% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.23% <ø> (+0.78%)` | :arrow_up: |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.06% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19206/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19206/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19206/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.80% <33.33%> (-0.05%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19206?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/cli/commands/TableCommand.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-cli%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcli%2Fcommands%2FTableCommand.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1RhYmxlQ29tbWFuZC5qYXZh) | `50.90% <ø> (ø)` | |
| [...main/java/org/apache/hudi/io/BaseCreateHandle.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FBaseCreateHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0Jhc2VDcmVhdGVIYW5kbGUuamF2YQ==) | `89.10% <100.00%> (+0.81%)` | :arrow_up: |
| [...ava/org/apache/hudi/io/HoodieWriteMergeHandle.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieWriteMergeHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlTWVyZ2VIYW5kbGUuamF2YQ==) | `87.55% <100.00%> (+0.18%)` | :arrow_up: |
| [...he/hudi/table/upgrade/NineToTenUpgradeHandler.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Ftable%2Fupgrade%2FNineToTenUpgradeHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL3VwZ3JhZGUvTmluZVRvVGVuVXBncmFkZUhhbmRsZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [.../hudi/io/storage/HoodieSparkFileWriterFactory.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2FHoodieSparkFileWriterFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTcGFya0ZpbGVXcml0ZXJGYWN0b3J5LmphdmE=) | `77.46% <100.00%> (+0.32%)` | :arrow_up: |
| [...rg/apache/hudi/HoodieDatasetBulkInsertHelper.scala](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieDatasetBulkInsertHelper.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZURhdGFzZXRCdWxrSW5zZXJ0SGVscGVyLnNjYWxh) | `92.36% <100.00%> (ø)` | |
| [...a/org/apache/hudi/common/model/MetaFieldsMode.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fmodel%2FMetaFieldsMode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL01ldGFGaWVsZHNNb2RlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2FHoodieTableConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `94.69% <100.00%> (+0.08%)` | :arrow_up: |
| [.../apache/hudi/common/table/TableSchemaResolver.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2FTableSchemaResolver.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL1RhYmxlU2NoZW1hUmVzb2x2ZXIuamF2YQ==) | `87.97% <100.00%> (+0.06%)` | :arrow_up: |
| [...io/storage/hadoop/HoodieAvroFileWriterFactory.java](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree&filepath=hudi-hadoop-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2Fstorage%2Fhadoop%2FHoodieAvroFileWriterFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1oYWRvb3AtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL3N0b3JhZ2UvaGFkb29wL0hvb2RpZUF2cm9GaWxlV3JpdGVyRmFjdG9yeS5qYXZh) | `93.10% <100.00%> (+0.12%)` | :arrow_up: |
| ... and [19 more](https://app.codecov.io/gh/apache/hudi/pull/19206?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [23 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19206/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5026946942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470)
* 4f5b3966ea0ed2ec7291ca64b2a364c362e47822 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5221293250
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179)
* e13ab243b73fae4568cfb976bec6b577090041f6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5221429012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e13ab243b73fae4568cfb976bec6b577090041f6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185)
* ed5b5263fa2b6a60abb79ccd696f892230130a7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-4897900952
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9775897985ad1fe221e957f75e7d549d57bddc7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3665334318
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllClearsProperty() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode so we have something to clear.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
Review Comment:
The result here is discarded. If this transition ever fails the table stays `ALL`, the `--target-mode ALL` on the next line hits the no-op early return (`TableCommand.java:292`), and all four assertions below still pass -- a freshly created table carries neither `populate.meta.fields` nor `meta.fields.mode`, so `getMetaFieldsMode()` is `ALL`, `populateMetaFields()` is `true`, and the key is absent.
Wrap this in `assertTrue(ShellEvaluationResultUtil.isSuccess(...))` and assert the property **is present** before clearing it, so the "was there -> now gone" transition is actually proven.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
Review Comment:
Accuracy point on "the sanctioned way": `table update-configs` sits 30 lines above and will set `hoodie.meta.fields.mode` from a props file with zero validation (`:228-244`) -- atomically, in a single modify cycle. `table delete-configs` and `repair overwrite-hoodie-props` are equally open.
Either move the transition check somewhere all writers go through, or have `update-configs` / `delete-configs` refuse this key and point here. Otherwise worth softening the claim in the PR description.
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
Review Comment:
Heads up before these are treated as the verification story: **`hudi-cli` tests do not run in CI.** The module appears only as an exclusion -- `azure-pipelines-20230430.yml:46,62` and `.github/workflows/bot.yml:242,244` -- and the install steps carry `-DskipTests` (`:109`). No job targets it.
Quick way to confirm: `testDefaultCreate:144` asserts timeline layout version 1, but `HoodieTableVersion.current()` is `TEN`, which maps to `LAYOUT_VERSION_2`. That assertion would be red if the suite ran.
So the green Azure run says nothing about these 9 tests. Suggest either adding `hudi-cli` to the CI module list in a companion change, or putting the load-bearing coverage somewhere CI actually exercises.
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllClearsProperty() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode so we have something to clear.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ // Then set back to ALL.
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode property should be cleared (ALL is implicit).
+ assertFalse(client.getTableConfig().getProps().containsKey(HoodieTableConfig.META_FIELDS_MODE.key()));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
Review Comment:
This passes even if the command is not registered at all.
`ShellEvaluationResultUtil.isSuccess` is just `!(x instanceof Throwable)`, and Spring Shell 2.1.1 *returns* a `CommandNotFound` (a `RuntimeException`) whose message echoes the input line verbatim. So a typo'd `@ShellMethod` key yields `No command found for 'table set-meta-fields-mode --target-mode BOGUS_MODE'`, which satisfies both assertions.
Assert on text only the production code can emit:
```suggestion
assertInstanceOf(HoodieException.class, result);
assertTrue(result.toString().contains("Unsupported --target-mode 'BOGUS_MODE'"),
"expected error message to name the rejected value, got: " + result);
```
(needs `org.junit.jupiter.api.Assertions.assertInstanceOf` and `org.apache.hudi.exception.HoodieException` imports)
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllClearsProperty() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode so we have something to clear.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ // Then set back to ALL.
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode property should be cleared (ALL is implicit).
+ assertFalse(client.getTableConfig().getProps().containsKey(HoodieTableConfig.META_FIELDS_MODE.key()));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
Review Comment:
Two things here.
The `||` is redundant slack -- I checked the plausible alternate failures and none satisfy it today, but there is no upside to keeping it.
More importantly, line 415 reads `getTableConfig()` off the metaclient snapshot taken *before* the command ran (`HoodieTableMetaClient` holds `tableConfig` as a plain field), and the command throws at `:298` before its final refresh. So this would read `ALL` even if the command had written to disk and then thrown -- it cannot catch a write-before-guard regression. `HoodieTableMetaClient.reload(...)` is the idiom used in `TestRepairsCommand.java:218`.
```suggestion
assertTrue(result.toString().contains(
"Refusing to change hoodie.meta.fields.mode on a table that already has 1 commit(s)"),
"expected refusal message, got: " + result);
// Mode must not have changed.
assertEquals(MetaFieldsMode.ALL,
HoodieTableMetaClient.reload(HoodieCLI.getTableMetaClient()).getTableConfig().getMetaFieldsMode());
```
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
Review Comment:
Two nits, feel free to ignore:
- `throws IOException` is dead -- `HoodieTableConfig.update/delete` wrap in unchecked `HoodieIOException`, and `refreshTableMetadata()` declares nothing.
- Lines 263 and 274 introduce the only non-ASCII characters in `hudi-cli/src/main` (em dashes, in CLI help text). Suggest plain `--`.
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllClearsProperty() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode so we have something to clear.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ // Then set back to ALL.
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode property should be cleared (ALL is implicit).
+ assertFalse(client.getTableConfig().getProps().containsKey(HoodieTableConfig.META_FIELDS_MODE.key()));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeWithForceOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
Review Comment:
These two assertions are a strict subset of `testSetMetaFieldsModeOnFreshTableToCommitTimeOnly`. The only thing making this a distinct scenario is that `createDummyCommitFile` produced a counted instant, and nothing here pins that -- if the fixture ever stops being recognised, this quietly becomes a duplicate of the fresh-table test and still passes green.
```suggestion
assertTrue(ShellEvaluationResultUtil.isSuccess(result));
// Pin the precondition: --force must be the branch actually under test.
assertEquals(1, HoodieCLI.getTableMetaClient().getActiveTimeline()
.getCommitsTimeline().countInstants());
assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
```
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllClearsProperty() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode so we have something to clear.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ // Then set back to ALL.
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode property should be cleared (ALL is implicit).
+ assertFalse(client.getTableConfig().getProps().containsKey(HoodieTableConfig.META_FIELDS_MODE.key()));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeWithForceOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ private void createDummyCommitFile(String instantTime) throws IOException {
+ // Timeline v2 layout: files live under .hoodie/timeline/. Writing an empty completed commit
Review Comment:
This hand-rolls the v2 timeline layout when the helper already exists and is already used in this same file at line 217: `HoodieTestDataGenerator.createCommitFile(tablePath, instantTime, storageConf())`. It is imported at line 37, and the sibling command's test does the same thing (`ITTestTableCommand.java:81-82`).
The helper also writes real commit metadata and a valid instant time -- `instantTime + "1"` here is an 18-char string that is not a valid Hudi timestamp.
Suggest deleting this method and calling the helper at both call sites (405 and 422). That also removes the fully-qualified `java.nio.file.*` references.
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,140 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
Review Comment:
nit, feel free to ignore: these three fresh-table tests differ only by the mode string. `@ParameterizedTest` + `@EnumSource(names = {"COMMIT_TIME_ONLY", "FILE_NAME_ONLY", "COMMIT_TIME_AND_FILE_NAME"})` would collapse them and let all three carry the full assertion set -- right now `FILE_NAME_ONLY` and `COMMIT_TIME_AND_FILE_NAME` skip the `assertFalse(populateMetaFields())` check the first one has. That check stops being implied once #19205's `resolve()` lands, since it no longer consults the boolean.
Keep `...ToNone` separate (it deletes the key rather than writing it), and do not hoist `prepareTable()` into `@BeforeEach` -- `testCreateWithSpecifiedValues` and `testRefresh` would start failing with "Table already existing".
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ targetMode = MetaFieldsMode.valueOf(targetModeStr.trim());
+ } catch (IllegalArgumentException e) {
+ throw new HoodieException(String.format(
+ "Unsupported --target-mode '%s'. Allowed values: ALL, NONE, COMMIT_TIME_ONLY, "
+ + "FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME.", targetModeStr));
+ }
+
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Map<String, String> oldProps = client.getTableConfig().propsMap();
+ MetaFieldsMode currentMode = client.getTableConfig().getMetaFieldsMode();
+
+ if (currentMode == targetMode) {
+ return String.format("Table is already in %s mode; nothing to change.", targetMode);
+ }
+
+ // Safety check: refuse to change the mode on a table with commits unless --force.
+ int commitCount = client.getActiveTimeline().getCommitsTimeline().countInstants();
Review Comment:
This lets you set a selective mode on a MoR table, and on a fresh table there are no commits so `--force` is not even needed. The writer already rejects that combination.
`HoodieWriteConfig.java:3936-3947` (added by this stack's own base commit `2b11a4fa`) throws for `MERGE_ON_READ + selective` and for `engine != SPARK + selective`. `HoodieSparkSqlWriter.scala:1102-1106` copies every table property into the write config, so once this command has written the mode, **every** subsequent write fails validation. The table is bricked and the CLI gave no warning.
Please reject selective modes up front when the table is MoR (or the writer is not Spark), reusing the writer's message. `--force` should not override this one -- it is an unimplemented code path, not a data-correctness tradeoff.
Worth a `create --tableType MERGE_ON_READ` case too; all 9 new tests go through `prepareTable()`, which is CoW only.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ targetMode = MetaFieldsMode.valueOf(targetModeStr.trim());
+ } catch (IllegalArgumentException e) {
+ throw new HoodieException(String.format(
+ "Unsupported --target-mode '%s'. Allowed values: ALL, NONE, COMMIT_TIME_ONLY, "
+ + "FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME.", targetModeStr));
+ }
+
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Map<String, String> oldProps = client.getTableConfig().propsMap();
+ MetaFieldsMode currentMode = client.getTableConfig().getMetaFieldsMode();
+
+ if (currentMode == targetMode) {
+ return String.format("Table is already in %s mode; nothing to change.", targetMode);
+ }
+
+ // Safety check: refuse to change the mode on a table with commits unless --force.
+ int commitCount = client.getActiveTimeline().getCommitsTimeline().countInstants();
+ if (commitCount > 0 && !force) {
+ throw new HoodieException(String.format(
+ "Refusing to change hoodie.meta.fields.mode on a table that already has %d commit(s). "
+ + "Existing files were written under %s and will not be rewritten by this command; "
+ + "new commits would be written under %s, producing mixed-mode files whose "
+ + "incremental / file-pruning semantics differ between old and new data. "
+ + "Pass --force if you accept the consequences, or recreate the table to change "
+ + "the mode cleanly.",
+ commitCount, currentMode, targetMode));
+ }
+ if (commitCount > 0) {
+ log.warn("--force passed: changing hoodie.meta.fields.mode from {} to {} on a table with "
Review Comment:
The warning undersells this. Widening (`COMMIT_TIME_ONLY -> ALL`, `NONE -> anything`) does not silently drop rows -- it throws.
Under a selective mode `_hoodie_record_key` is a physically present NULL column. Once this sets `populate.meta.fields=true`, `RecordContext.java:76-78` switches to `metadataKeyExtractor()`, which is an unconditional `.toString()` on that null (`RecordContext.java:438-439`). Every read of a pre-existing file group NPEs.
It is also the one way to defeat the guard permanently: `BaseHoodieWriteClient.java:1552` compares *writer vs table* config, so rewriting the table config means it can never fire again. That invariant goes back to `d5026e9a2485` (HUDI-2161, 2021), and #19205 restates it per-mode via `isWiderThan`.
Suggest making widening a hard refusal that `--force` cannot override, and keeping `--force` for narrowing only.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ targetMode = MetaFieldsMode.valueOf(targetModeStr.trim());
+ } catch (IllegalArgumentException e) {
+ throw new HoodieException(String.format(
+ "Unsupported --target-mode '%s'. Allowed values: ALL, NONE, COMMIT_TIME_ONLY, "
+ + "FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME.", targetModeStr));
+ }
+
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Map<String, String> oldProps = client.getTableConfig().propsMap();
+ MetaFieldsMode currentMode = client.getTableConfig().getMetaFieldsMode();
+
+ if (currentMode == targetMode) {
+ return String.format("Table is already in %s mode; nothing to change.", targetMode);
+ }
+
+ // Safety check: refuse to change the mode on a table with commits unless --force.
+ int commitCount = client.getActiveTimeline().getCommitsTimeline().countInstants();
+ if (commitCount > 0 && !force) {
+ throw new HoodieException(String.format(
+ "Refusing to change hoodie.meta.fields.mode on a table that already has %d commit(s). "
+ + "Existing files were written under %s and will not be rewritten by this command; "
+ + "new commits would be written under %s, producing mixed-mode files whose "
+ + "incremental / file-pruning semantics differ between old and new data. "
+ + "Pass --force if you accept the consequences, or recreate the table to change "
+ + "the mode cleanly.",
+ commitCount, currentMode, targetMode));
+ }
+ if (commitCount > 0) {
+ log.warn("--force passed: changing hoodie.meta.fields.mode from {} to {} on a table with "
+ + "{} commit(s). Existing files retain the old layout; new commits use the new "
+ + "layout. Incremental queries and file-name lookups may silently drop rows written "
+ + "under the incompatible mode.",
+ currentMode, targetMode, commitCount);
+ }
+
+ // Persist. ALL and NONE are implicit (derived from populate.meta.fields); selective modes are
+ // written explicitly. We also toggle populate.meta.fields to keep both properties in sync so
+ // older readers that only understand the legacy boolean still degrade correctly.
+ Properties toUpdate = new Properties();
+ toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(),
+ String.valueOf(targetMode == MetaFieldsMode.ALL));
+ if (targetMode == MetaFieldsMode.ALL || targetMode == MetaFieldsMode.NONE) {
+ // Selective mode is being cleared; delete the property rather than write empty string.
+ HoodieTableConfig.delete(client.getStorage(), client.getMetaPath(),
+ Collections.singleton(HoodieTableConfig.META_FIELDS_MODE.key()));
+ HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate);
+ } else {
+ toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(), targetMode.name());
+ HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate);
+ }
Review Comment:
These are two independent `modify()` cycles (`HoodieTableConfig.java:552-600`), and the state between them is `NONE` -- meta fields off entirely, worse than either endpoint. If the second call fails the table stays there: `modify()` drops its backup on success, and `recoverIfNeeded` only kicks in when `hoodie.properties` is missing or invalid, so `table recover-configs` cannot undo it.
That breaks the one-transaction-per-mutation invariant from `ae0c67d9fc7c` (HUDI-2795). `updateAndDeleteProps` already exists for exactly this "set some, delete some" shape (`HoodieTableConfig.java:616`, added by `b60d38c40fa1`).
```suggestion
Properties toUpdate = new Properties();
toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(),
String.valueOf(targetMode == MetaFieldsMode.ALL));
boolean implicitMode = targetMode == MetaFieldsMode.ALL || targetMode == MetaFieldsMode.NONE;
if (!implicitMode) {
toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(), targetMode.name());
}
// Single modify() cycle: never leave the table in an intermediate mode.
HoodieTableConfig.updateAndDeleteProps(client.getStorage(), client.getMetaPath(), toUpdate,
implicitMode ? Collections.singleton(HoodieTableConfig.META_FIELDS_MODE.key())
: Collections.emptySet());
```
If you take the rebase fix in my other comment, the delete goes away entirely and this collapses to a plain `update()`.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ targetMode = MetaFieldsMode.valueOf(targetModeStr.trim());
Review Comment:
`valueOf` is case-sensitive, so `--target-mode all` fails. #19205 added `MetaFieldsMode.parse()` (case-insensitive, `7dcd9d1b`) at danny0405's request precisely for user-supplied input -- and the CLI is the most user-facing input surface we have. After the rebase this would be the only entry point still rejecting lowercase.
There are also three different "allowed values" lists now: this one, the enum's (which omits `ALL`), and the config doc at `HoodieTableConfig.java:340-342`. Switching to `MetaFieldsMode.parse(targetModeStr)` and deleting the local try/catch collapses them to one.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java:
##########
@@ -256,6 +258,82 @@ public String deleteTableConfig(
return renderOldNewProps(newProps, oldProps);
}
+ @ShellMethod(key = "table set-meta-fields-mode",
+ value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a "
+ + "table that already has commits unless --force is passed — the property is a "
+ + "physical-storage decision baked into files at write time, so mixing modes across "
+ + "commits produces mixed-mode files whose incremental / file-pruning semantics differ "
+ + "between old and new data. Use only on a table with zero commits, or with --force if "
+ + "you accept the data-correctness consequences.")
+ public String setMetaFieldsMode(
+ @ShellOption(value = {"--target-mode"},
+ help = "One of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME")
+ final String targetModeStr,
+ @ShellOption(value = {"--force"}, defaultValue = "false",
+ help = "Override the safety check that prevents changing the mode on a table with commits. "
+ + "Existing files are not rewritten — new commits use the new mode, old commits keep "
+ + "the old mode. Incremental queries and file-name-based lookups will silently drop "
+ + "rows from commits written under the incompatible mode.")
+ final boolean force) throws IOException {
+ MetaFieldsMode targetMode;
+ try {
+ targetMode = MetaFieldsMode.valueOf(targetModeStr.trim());
+ } catch (IllegalArgumentException e) {
+ throw new HoodieException(String.format(
+ "Unsupported --target-mode '%s'. Allowed values: ALL, NONE, COMMIT_TIME_ONLY, "
+ + "FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME.", targetModeStr));
+ }
+
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Map<String, String> oldProps = client.getTableConfig().propsMap();
+ MetaFieldsMode currentMode = client.getTableConfig().getMetaFieldsMode();
+
+ if (currentMode == targetMode) {
+ return String.format("Table is already in %s mode; nothing to change.", targetMode);
+ }
+
+ // Safety check: refuse to change the mode on a table with commits unless --force.
+ int commitCount = client.getActiveTimeline().getCommitsTimeline().countInstants();
+ if (commitCount > 0 && !force) {
+ throw new HoodieException(String.format(
+ "Refusing to change hoodie.meta.fields.mode on a table that already has %d commit(s). "
+ + "Existing files were written under %s and will not be rewritten by this command; "
+ + "new commits would be written under %s, producing mixed-mode files whose "
+ + "incremental / file-pruning semantics differ between old and new data. "
+ + "Pass --force if you accept the consequences, or recreate the table to change "
+ + "the mode cleanly.",
+ commitCount, currentMode, targetMode));
+ }
+ if (commitCount > 0) {
+ log.warn("--force passed: changing hoodie.meta.fields.mode from {} to {} on a table with "
+ + "{} commit(s). Existing files retain the old layout; new commits use the new "
+ + "layout. Incremental queries and file-name lookups may silently drop rows written "
+ + "under the incompatible mode.",
+ currentMode, targetMode, commitCount);
+ }
+
+ // Persist. ALL and NONE are implicit (derived from populate.meta.fields); selective modes are
Review Comment:
This PR is stacked on a snapshot of #19205 that is now 9 commits behind, and it encodes the semantics #19205 has since dropped.
`c78dc962` + `664ff2ec` made `hoodie.meta.fields.mode` the source of truth, persisted verbatim including `ALL` and `NONE`, with the legacy boolean derived from it -- that was the fix for danny0405's P1. And `4134dac6`'s `NineToTenUpgradeHandler` now writes the property explicitly onto exactly the tables this command would strip it from.
Please rebase and drop the clearing branch: always write `META_FIELDS_MODE=targetMode.name()` plus the derived boolean, in one `update()`. The assertion at `TestTableCommand.java:374` pins the old representation and will need inverting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5337846291
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
}, {
"hash" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 097c8d95b3abfc325314c7cc053e113ec014f725 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255)
* 0f15545398cbb26bbaa6ef652e67e0778bb9f366 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5271334917
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 097c8d95b3abfc325314c7cc053e113ec014f725 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5219365673
Rebased onto #19205's current head, and tightened the command to match the rule that PR establishes.
Previously this branch was stacked on `13b39db9fb22`, 17 commits behind. It now sits on top of all of #19205, so the two read as a clean stack: **#19205 makes `hoodie.meta.fields.mode` a table property a write can never change; this PR provides the only sanctioned way to change it.**
### Three gaps closed against that rule
**1. Widening was allowed.** The `--force` gate keyed only on commit count, so `--target-mode ALL --force` on a populated `COMMIT_TIME_ONLY` table went through. Existing files are not rewritten, so the table would advertise a meta column that is physically null for every earlier row — and incremental queries would then admit the table and silently skip exactly those rows.
Widening is now **refused outright, and `--force` does not override it**. The asymmetry is deliberate:
- *Narrowing* leaves earlier files carrying values nothing reads. A reader can ignore them, so `--force` remains the right gate — the operator is accepting mixed-mode files.
- *Widening* leaves later files claiming a column earlier files do not have, with nothing to distinguish the two sets. There is no consequence an operator can knowingly accept, so it is a hard failure.
This uses `MetaFieldsMode#isWiderThan` — the same predicate `BaseHoodieWriteClient#validateAgainstTableProperties` uses in #19205 — so the CLI and the write path cannot drift on which transitions are legal.
**2. `ALL` and `NONE` deleted the mode property rather than writing it.** Three problems: the table then resolved through the legacy fallback, indistinguishable from a table predating the property; the command's effect became invisible to anyone reading `hoodie.properties`; and `TenToNineDowngradeHandler` reads the mode to derive the boolean it writes back on downgrade. Both properties are now always written, with the boolean **derived** from the mode — the same invariant `HoodieTableMetaClient.TableBuilder` enforces at creation, so `hoodie.properties` can never contradict itself.
**3. `valueOf` rejected lowercase and produced a bare `No enum constant`.** Routed through `MetaFieldsMode.parse`, so `--target-mode commit_time_only` works and the error lists the allowed values like every other path that parses this property.
### Tests
Five new cases: widening refused in three forms (including both `--force` variants), narrowing refused without `--force` and accepted with it, lowercase accepted, and both properties in agreement across all five modes.
`testSetMetaFieldsModeToAllClearsProperty` asserted the deleted-property behavior from gap 2, so it is rewritten as `testSetMetaFieldsModeToAllWritesTheModeExplicitly` rather than deleted — the old name pinned a decision that turned out to be wrong, and the new one pins the corrected invariant.
### Note on local build
`hudi-cli` compilation is currently blocked locally by a **stale `hudi-utilities-bundle`** that embeds an older shaded `MetaFieldsMode` and shadows `hudi-common` on the classpath; rebuilding that bundle fails on an unrelated pre-existing error in `KafkaAvroSchemaDeserializer` (reproduces on `master`). I verified `TableCommand.java` and `TestTableCommand.java` compile cleanly against a classpath with the stale bundle excluded, and checkstyle passes on `hudi-cli`. CI builds everything fresh so it should not hit this, but flagging it in case anyone sees the same locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5219462991
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f5b3966ea0ed2ec7291ca64b2a364c362e47822 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553)
* 4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5270598248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195)
* 097c8d95b3abfc325314c7cc053e113ec014f725 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3743362178
##########
hudi-common/src/main/java/org/apache/hudi/common/model/MetaFieldsMode.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.Locale;
+
+/**
+ * Which of Hudi's meta columns are physically populated on disk.
+ *
+ * <p>Selective modes exist so that tables that opt out of the default {@code populate.meta.fields=true}
+ * can still keep the two columns that matter for downstream operations without paying for the other
+ * three:
+ *
+ * <ul>
+ * <li>{@code _hoodie_commit_time} — required for incremental queries.</li>
+ * <li>{@code _hoodie_file_name} — useful for file-level pruning / investigation lookups.</li>
+ * </ul>
+ *
+ * <p>The remaining three meta columns ({@code _hoodie_commit_seqno}, {@code _hoodie_record_key},
+ * {@code _hoodie_partition_path}) are all-or-nothing — either populate every meta column ({@link #ALL})
+ * or none of them beyond the two selectable ones. If you need any of the remaining columns, set
+ * {@code hoodie.populate.meta.fields=true}.
+ *
+ * <p>This enum is the single source of truth for meta-column population. The legacy boolean
+ * {@code hoodie.populate.meta.fields} is deprecated and consulted only when
+ * {@code hoodie.meta.fields.mode} is absent, so that tables written before the mode property
+ * existed keep their behavior:
+ *
+ * <ul>
+ * <li>{@code populate.meta.fields=true} (or absent) → {@link #ALL} — today's default.</li>
+ * <li>{@code populate.meta.fields=false} → {@link #NONE}.</li>
+ * </ul>
+ *
+ * <p>On-disk representation: the enum {@link #name()} is persisted in {@code hoodie.properties}
+ * under the property {@code hoodie.meta.fields.mode}.
+ */
+public enum MetaFieldsMode {
+ /**
+ * All five Hudi meta columns are populated — today's default.
+ */
+ ALL(true, true),
+
+ /**
+ * No Hudi meta columns are populated. Incremental queries are unsupported. File-level pruning
+ * that depends on {@code _hoodie_file_name} is unsupported.
+ */
+ NONE(false, false),
+
+ /**
+ * Only {@code _hoodie_commit_time} is populated. Incremental queries remain functional; other
+ * meta columns stay null on disk.
+ */
+ COMMIT_TIME_ONLY(true, false),
+
+ /**
+ * Only {@code _hoodie_file_name} is populated. Useful for file-level lookups and debugging;
+ * incremental queries are unsupported.
+ */
+ FILE_NAME_ONLY(false, true),
+
+ /**
+ * Both {@code _hoodie_commit_time} and {@code _hoodie_file_name} are populated.
+ */
+ COMMIT_TIME_AND_FILE_NAME(true, true);
+
+ private final boolean commitTimePopulated;
+ private final boolean fileNamePopulated;
+
+ MetaFieldsMode(boolean commitTimePopulated, boolean fileNamePopulated) {
+ this.commitTimePopulated = commitTimePopulated;
+ this.fileNamePopulated = fileNamePopulated;
+ }
+
+ public boolean isCommitTimePopulated() {
+ return commitTimePopulated;
+ }
+
+ public boolean isFileNamePopulated() {
+ return fileNamePopulated;
+ }
+
+ /**
+ * @return true when all five meta columns are populated (i.e. this is {@link #ALL}). Selective
+ * modes never populate {@code _hoodie_record_key}, {@code _hoodie_partition_path}, or
+ * {@code _hoodie_commit_seqno}.
+ */
+ public boolean isRecordKeyPopulated() {
+ return this == ALL;
+ }
+
+ /**
+ * @return true for the modes that populate some but not all meta columns, i.e. everything except
+ * {@link #ALL} and {@link #NONE}.
+ *
+ * <p>These are the modes the deprecated {@code hoodie.populate.meta.fields} boolean cannot
+ * express, so they are what callers gate on when a code path only understands all-or-nothing meta
+ * fields — writer engines not yet wired for selective population, table versions that predate the
+ * mode property, and validation that must not let a two-state writer speak for a five-state table.
+ */
+ public boolean isSelective() {
+ return this != ALL && this != NONE;
+ }
+
+ /**
+ * Resolve the effective mode from any {@link HoodieConfig} that may carry the two properties —
+ * a table config, a write config, or a bare config built from write options. Preferred over the
+ * two-argument overload: it keeps the property keys and the precedence rule in one place instead
+ * of repeating them at every call site.
+ */
+ public static MetaFieldsMode resolve(HoodieConfig config) {
+ return resolve(config.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE),
+ config.getBooleanOrDefault(HoodieTableConfig.POPULATE_META_FIELDS));
+ }
+
+ /**
+ * Resolve the effective mode. {@code hoodie.meta.fields.mode} is the source of truth; the
+ * deprecated {@code hoodie.populate.meta.fields} boolean is a fallback for tables written before
+ * the mode property existed. Precedence:
+ *
+ * <ul>
+ * <li>non-empty mode → the parsed enum value (the legacy boolean is not consulted).</li>
+ * <li>null/empty mode + {@code populateMetaFields=false} → {@link #NONE}.</li>
+ * <li>null/empty mode + {@code populateMetaFields=true} → {@link #ALL}.</li>
+ * </ul>
+ *
+ * @param rawMode raw {@code hoodie.meta.fields.mode} value; may be null or empty.
+ * @param legacyPopulateMetaFields value of the deprecated {@code hoodie.populate.meta.fields}.
+ * @throws IllegalArgumentException when the raw mode value does not match any enum value. This
+ * includes the pre-enum comma-separated format — callers that upgrade an old table must
+ * migrate the value through the hudi-cli.
+ */
+ public static MetaFieldsMode resolve(String rawMode, boolean legacyPopulateMetaFields) {
+ if (StringUtils.isNullOrEmpty(rawMode)) {
+ return legacyPopulateMetaFields ? ALL : NONE;
+ }
+ return parse(rawMode);
+ }
+
+ /**
+ * Parse a raw {@code hoodie.meta.fields.mode} value into an enum constant, with a message that
+ * lists the allowed values. Prefer this over {@link #valueOf(String)} for user-supplied input.
+ */
+ public static MetaFieldsMode parse(String rawMode) {
+ try {
+ // Case-insensitive: users hand-editing hoodie.properties or passing write options should not
Review Comment:
🤖 nit: could you derive the allowed-values list dynamically here instead of hardcoding it? Something like `Arrays.stream(values()).map(Enum::name).collect(Collectors.joining(", "))` — that way the message stays accurate if a new constant is added later without anyone remembering to update this string.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +305,316 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllWritesTheModeExplicitly() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode, then back to ALL. Legal here only because the table has no
+ // commits — on a populated table this would be a widening and refused outright.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode is written explicitly rather than deleted. Deleting it would leave the table
+ // resolving through the legacy fallback -- indistinguishable from a table predating the
+ // property -- and would make this command's effect invisible in hoodie.properties. The v10->v9
+ // downgrade handler also reads the mode to derive the boolean it writes back.
+ assertEquals(MetaFieldsMode.ALL.name(),
+ client.getTableConfig().getString(HoodieTableConfig.META_FIELDS_MODE));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesWideningOnPopulatedTableEvenWithForce() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Widening is one-way-forbidden and --force does not override it: existing files are not
+ // rewritten, so the table would advertise a column that is null for every row written so far,
+ // and incremental queries would then admit the table and silently skip those rows. There is no
+ // consequence for an operator to knowingly accept, so this is a hard failure.
+ for (String command : new String[] {
+ "table set-meta-fields-mode --target-mode ALL",
+ "table set-meta-fields-mode --target-mode ALL --force true",
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME --force true"}) {
+ Object result = shell.evaluate(() -> command);
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result), "expected refusal for: " + command);
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for '" + command + "', got: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "mode must be unchanged after a refused widening");
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAllowsNarrowingOnPopulatedTableWithForce() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Narrowing is the direction the CLI exists to allow. It still needs --force, because it leaves
+ // mixed-mode files, but it is not refused outright the way widening is.
+ Object refused = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(refused));
+
+ Object forced = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(forced), "narrowing with --force must succeed: " + forced);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAcceptsLowercaseTargetMode() {
+ assertTrue(prepareTable());
+ // Routed through MetaFieldsMode.parse rather than valueOf, so an operator typing the mode in
+ // lower case is not rejected.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode commit_time_only");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result), "expected lowercase to be accepted: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeKeepsBothPropertiesInAgreement() {
+ assertTrue(prepareTable());
+ // hoodie.properties must never contradict itself: the legacy boolean is derived from the mode,
+ // never taken from the caller, so a pre-1.3.0 reader that sees only the boolean treats a
+ // selective table as NONE rather than assuming meta columns that are physically null.
+ for (MetaFieldsMode mode : MetaFieldsMode.values()) {
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " + mode.name() + " --force true");
+ HoodieTableConfig tableConfig = HoodieCLI.getTableMetaClient().getTableConfig();
+ if (tableConfig.getMetaFieldsMode() == mode) {
+ assertEquals(mode.toLegacyPopulateMetaFields(), tableConfig.populateMetaFields(),
+ "populate.meta.fields must be the derived value for mode " + mode);
+ }
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeWithForceOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ /**
+ * Pins the whole migration matrix rather than spot-checking it. Twenty ordered pairs, each either
+ * a legal narrowing or a refused widening, on a table that already has commits.
+ *
+ * <p>The subtle entries are the mutually-wider siblings: {@code COMMIT_TIME_ONLY} and
+ * {@code FILE_NAME_ONLY} each populate a column the other does not, so neither can migrate to the
+ * other in either direction. A spot-check of "narrowing works, widening does not" would miss that
+ * the relation is a lattice rather than a chain.
+ *
+ * <p>Uses a fresh table per pair so the starting mode can be set without tripping the very guard
+ * under test -- setting the initial mode happens before any commit exists.
+ */
+ @Test
+ public void testSetMetaFieldsModeMigrationMatrixOnPopulatedTable() throws Exception {
+ for (MetaFieldsMode from : MetaFieldsMode.values()) {
+ for (MetaFieldsMode to : MetaFieldsMode.values()) {
+ if (from == to) {
+ continue;
+ }
+ // Fresh table per pair; connect to it so HoodieCLI points at the right one.
+ String pairName = tableName + "_" + from.name() + "_to_" + to.name();
+ String pairPath = tablePath(pairName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + pairPath + " --tableName " + pairName)));
+
+ // Establish the starting mode while the table is still empty, then make it "populated".
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " + from.name())),
+ "setting the initial mode on an empty table must succeed: " + from);
+ createDummyCommitFileAt(pairPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+ assertEquals(from, HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ boolean widening = to.isWiderThan(from);
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode " + to.name() + " --force true");
+
+ if (widening) {
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " adds a meta column and must be refused even with --force");
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for " + from + " -> " + to + ", got: " + result);
+ assertEquals(from, HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "a refused migration must leave the mode untouched: " + from + " -> " + to);
+ } else {
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " drops meta columns and must be allowed with --force, got: " + result);
+ HoodieTableConfig tableConfig = HoodieCLI.getTableMetaClient().getTableConfig();
+ assertEquals(to, tableConfig.getMetaFieldsMode(), from + " -> " + to);
+ assertEquals(to.toLegacyPopulateMetaFields(), tableConfig.populateMetaFields(),
+ "the derived boolean must follow the new mode for " + from + " -> " + to);
+ }
+ }
+ }
+ }
+
+ /** Both mutually-wider directions between the two single-column modes are refused. */
+ @Test
+ public void testSetMetaFieldsModeRefusesBothSiblingDirections() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toSibling = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY --force true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toSibling),
+ "COMMIT_TIME_ONLY -> FILE_NAME_ONLY adds _hoodie_file_name and must be refused");
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ // ...and the reverse, on a table that starts the other way round.
+ String otherName = tableName + "_sibling_reverse";
+ String otherPath = tablePath(otherName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + otherPath + " --tableName " + otherName)));
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ createDummyCommitFileAt(otherPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toOther = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toOther),
+ "FILE_NAME_ONLY -> COMMIT_TIME_ONLY adds _hoodie_commit_time and must be refused");
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ private void createDummyCommitFileAt(String tableBasePath, String instantTime) throws IOException {
+ java.nio.file.Path timelineDir =
+ Paths.get(tableBasePath, METAFOLDER_NAME, "timeline");
+ if (!timelineDir.toFile().exists()) {
+ timelineDir.toFile().mkdirs();
+ }
+ String completionTime = instantTime + "1";
+ java.nio.file.Files.createFile(timelineDir.resolve(instantTime + "_" + completionTime + ".commit"));
Review Comment:
🤖 nit: `createDummyCommitFile` and `createDummyCommitFileAt` have nearly identical bodies — could `createDummyCommitFile` just delegate to `createDummyCommitFileAt` by passing the table base path derived from `metaPath`? Keeps the file-creation logic in one place.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieStreamerMetaFieldsMode.java:
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.MetaFieldsMode;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.functions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * End-to-end coverage for {@code hoodie.meta.fields.mode} through the HoodieStreamer entrypoint.
+ * Each parameterized invocation runs a single ingest cycle in the given {@link MetaFieldsMode} and
+ * verifies both the persisted table property and the actual on-disk parquet column population.
+ *
+ * <p>Rejection paths (unknown token, populate=true+mode, MoR+mode) are exercised in the datasource
+ * test {@code TestMetaFieldsMode}; this fixture focuses on the streamer control-flow.
+ */
+public class TestHoodieStreamerMetaFieldsMode extends HoodieDeltaStreamerTestBase {
+
+ /**
+ * Only the selective modes are parameterized here. ALL and NONE add no mode key at all, so they
+ * exercise none of the streamer-side plumbing this feature introduced — they are covered by the
+ * datasource tests and by {@code TestHoodieTableConfig}'s resolution cases.
+ */
+ @ParameterizedTest
+ @EnumSource(value = MetaFieldsMode.class,
+ names = {"COMMIT_TIME_ONLY", "FILE_NAME_ONLY", "COMMIT_TIME_AND_FILE_NAME"})
+ public void testStreamerRespectsMetaFieldsMode(MetaFieldsMode mode) throws Exception {
+ String tablePath = basePath + "/streamer_meta_fields_mode_" + mode.name();
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ // Force CoW; selective modes are CoW-only until MoR log-write is wired.
+ cfg.tableType = "COPY_ON_WRITE";
+ // The mode alone — pairing it with populate.meta.fields would now be a stated conflict, since the
+ // mode is authoritative and the boolean is only the fallback for resolving an absent one.
+ cfg.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + mode.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(cfg, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(mode, metaClient.getTableConfig().getMetaFieldsMode(),
+ "streamer must persist mode=" + mode + " on hoodie.properties");
+ assertOnDiskMetaColumns(tablePath, mode);
+ }
+
+ /**
+ * The regression this fixture exists for, and the one cshuo raised in review: a restarted streamer
+ * that does not restate the mode must keep writing the table's meta columns.
+ *
+ * <p>StreamSync builds its write config from {@code props} alone, and the mode is persisted only by
+ * {@code initializeEmptyTable}, which runs solely when the base path does not exist. So a second
+ * run used to resolve to {@link MetaFieldsMode#NONE} and write base files with a null
+ * {@code _hoodie_commit_time} while {@code hoodie.properties} still advertised
+ * {@code COMMIT_TIME_ONLY} — incremental queries were then admitted and silently dropped every one
+ * of those rows.
+ *
+ * <p>The rule now lives in {@code BaseHoodieWriteClient#validateAgainstTableProperties} for every
+ * engine rather than in StreamSync, so this asserts it end-to-end through the streamer: a run that
+ * states neither meta-field property inherits the table's mode.
+ */
+ @Test
+ public void testRestartWithoutRestatingTheModeKeepsWritingCommitTimes() throws Exception {
+ String tablePath = basePath + "/streamer_restart_inherits_mode";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieTestUtils.createMetaClient(context, tablePath).getTableConfig().getMetaFieldsMode());
+
+ // Restart against the existing table stating neither the mode nor the legacy boolean.
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "the restart must not have changed the table's mode");
+ assertTrue(metaClient.getActiveTimeline().filterCompletedInstants().countInstants() >= 2,
+ "expected the restart to have produced a second commit");
+
+ // Every row across both commits carries a commit time. A row with a null one is what incremental
+ // queries silently drop, so this is the assertion that catches the regression.
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0,
+ raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time on a COMMIT_TIME_ONLY table");
+ assertTrue(raw.select(HoodieRecord.COMMIT_TIME_METADATA_FIELD).distinct().count() >= 2,
+ "both commits must be represented, so the second run really did write through this path");
+ }
+
+ /**
+ * The variant @voonhous asked for, and the one cshuo originally described: the restart states the
+ * deprecated boolean rather than nothing at all.
+ *
+ * <p>These are different cases under the current rule. Stating neither meta-field property inherits
+ * the table's mode (above); stating the boolean is an explicit request that contradicts a
+ * {@code COMMIT_TIME_ONLY} table, so it is rejected rather than silently narrowing the write to
+ * {@code NONE}. Before this rule it narrowed silently, writing base files with a null
+ * {@code _hoodie_commit_time} into a table that still advertised the mode.
+ */
+ @Test
+ public void testRestartStatingTheLegacyBooleanIsRejected() throws Exception {
+ String tablePath = basePath + "/streamer_restart_legacy_boolean_conflict";
+
+ HoodieDeltaStreamer.Config first = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ first.tableType = "COPY_ON_WRITE";
+ first.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(first, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+
+ HoodieDeltaStreamer.Config restart = TestHelpers.makeConfig(tablePath, WriteOperationType.INSERT);
+ restart.tableType = "COPY_ON_WRITE";
+ restart.configs.add(HoodieTableConfig.POPULATE_META_FIELDS.key() + "=false");
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer restarted = new HoodieDeltaStreamer(restart, jsc);
+ restarted.getIngestionService().ingestOnce();
+ restarted.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key())
+ || rootMessage.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected a meta-fields conflict, got: " + rootMessage);
+
+ // The failed run must not have changed the table, nor written rows with a null commit time.
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(context, tablePath);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, metaClient.getTableConfig().getMetaFieldsMode(),
+ "a rejected restart must leave the table's mode untouched");
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ assertEquals(0, raw.filter(functions.col(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isNull()).count(),
+ "no row may have a null _hoodie_commit_time");
+ }
+
+ @Test
+ public void testStreamerRejectsMorWithSelectiveMode() throws Exception {
+ String tablePath = basePath + "/streamer_mor_selective_rejected";
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tablePath, WriteOperationType.BULK_INSERT);
+ cfg.tableType = "MERGE_ON_READ";
+ cfg.configs.add(HoodieTableConfig.META_FIELDS_MODE.key() + "=" + MetaFieldsMode.COMMIT_TIME_ONLY.name());
+
+ Throwable thrown = assertThrows(Throwable.class, () -> {
+ HoodieDeltaStreamer streamer = new HoodieDeltaStreamer(cfg, jsc);
+ streamer.getIngestionService().ingestOnce();
+ streamer.shutdownGracefully();
+ });
+
+ String rootMessage = rootMessageOf(thrown);
+ assertTrue(rootMessage.contains("COPY_ON_WRITE") || rootMessage.contains("MERGE_ON_READ")
+ || rootMessage.contains("MoR") || rootMessage.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "Expected MoR-restriction error, got: " + rootMessage);
+ }
+
+ private void assertOnDiskMetaColumns(String tablePath, MetaFieldsMode expectedMode) {
+ // Default HoodieTestDataGenerator partitions are YYYY/MM/DD (three levels).
+ Dataset<Row> raw = sparkSession.read().parquet(tablePath + "/*/*/*/*.parquet");
+ Row first = raw.select(
+ HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+ HoodieRecord.COMMIT_SEQNO_METADATA_FIELD,
+ HoodieRecord.RECORD_KEY_METADATA_FIELD,
+ HoodieRecord.PARTITION_PATH_METADATA_FIELD,
+ HoodieRecord.FILENAME_METADATA_FIELD).first();
+
+ if (expectedMode.isCommitTimePopulated()) {
+ assertNotNull(first.get(0), "commit_time must be populated in mode " + expectedMode);
Review Comment:
🤖 nit: could you use `first.getAs(HoodieRecord.COMMIT_TIME_METADATA_FIELD)` (and similarly for indices 1–4) instead of positional `get(0)` / `get(4)`? The magic indices are only correct as long as the `select()` call stays in exactly this order, so a future reorder would silently test the wrong column.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-4897865034
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9775897985ad1fe221e957f75e7d549d57bddc7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5270801706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195)
* 097c8d95b3abfc325314c7cc053e113ec014f725 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5337883525
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
}, {
"hash" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16376",
"triggerID" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 097c8d95b3abfc325314c7cc053e113ec014f725 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255)
* 0f15545398cbb26bbaa6ef652e67e0778bb9f366 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16376)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3810296261
##########
hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestTableCommand.java:
##########
@@ -304,4 +307,316 @@ private String getFileContent(String fileToReadStr) throws IOException {
fis.close();
return fromUTF8Bytes(data);
}
+
+ // ---------------------------------------------------------------------------
+ // set-meta-fields-mode
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCommitTimeOnly() {
+ assertTrue(prepareTable());
+ // Default table is ALL — no commits yet, so the safety check must let this through.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ // Rendered diff must surface the changed properties so the operator can confirm the write.
+ String rendered = result.toString();
+ assertTrue(rendered.contains(HoodieTableConfig.POPULATE_META_FIELDS.key()),
+ "expected rendered diff to mention populate.meta.fields, got: " + rendered);
+ assertTrue(rendered.contains(HoodieTableConfig.META_FIELDS_MODE.key()),
+ "expected rendered diff to mention meta.fields.mode, got: " + rendered);
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToFileNameOnly() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToCombinedMode() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_AND_FILE_NAME,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeOnFreshTableToNone() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode NONE");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.NONE, client.getTableConfig().getMetaFieldsMode());
+ assertFalse(client.getTableConfig().populateMetaFields());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeToAllWritesTheModeExplicitly() throws IOException {
+ assertTrue(prepareTable());
+ // First move to a selective mode, then back to ALL. Legal here only because the table has no
+ // commits — on a populated table this would be a widening and refused outright.
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ Object result = shell.evaluate(() -> "table set-meta-fields-mode --target-mode ALL");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ assertEquals(MetaFieldsMode.ALL, client.getTableConfig().getMetaFieldsMode());
+ assertTrue(client.getTableConfig().populateMetaFields());
+ // The mode is written explicitly rather than deleted. Deleting it would leave the table
+ // resolving through the legacy fallback -- indistinguishable from a table predating the
+ // property -- and would make this command's effect invisible in hoodie.properties. The v10->v9
+ // downgrade handler also reads the mode to derive the boolean it writes back.
+ assertEquals(MetaFieldsMode.ALL.name(),
+ client.getTableConfig().getString(HoodieTableConfig.META_FIELDS_MODE));
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesWideningOnPopulatedTableEvenWithForce() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Widening is one-way-forbidden and --force does not override it: existing files are not
+ // rewritten, so the table would advertise a column that is null for every row written so far,
+ // and incremental queries would then admit the table and silently skip those rows. There is no
+ // consequence for an operator to knowingly accept, so this is a hard failure.
+ for (String command : new String[] {
+ "table set-meta-fields-mode --target-mode ALL",
+ "table set-meta-fields-mode --target-mode ALL --force true",
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME --force true"}) {
+ Object result = shell.evaluate(() -> command);
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result), "expected refusal for: " + command);
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for '" + command + "', got: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "mode must be unchanged after a refused widening");
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAllowsNarrowingOnPopulatedTableWithForce() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_AND_FILE_NAME");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ // Narrowing is the direction the CLI exists to allow. It still needs --force, because it leaves
+ // mixed-mode files, but it is not refused outright the way widening is.
+ Object refused = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(refused));
+
+ Object forced = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(forced), "narrowing with --force must succeed: " + forced);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeAcceptsLowercaseTargetMode() {
+ assertTrue(prepareTable());
+ // Routed through MetaFieldsMode.resolve rather than valueOf, so an operator typing the mode in
+ // lower case is not rejected.
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode commit_time_only");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result), "expected lowercase to be accepted: " + result);
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeKeepsBothPropertiesInAgreement() {
+ assertTrue(prepareTable());
+ // hoodie.properties must never contradict itself: the legacy boolean is derived from the mode,
+ // never taken from the caller, so a pre-1.3.0 reader that sees only the boolean treats a
+ // selective table as NONE rather than assuming meta columns that are physically null.
+ for (MetaFieldsMode mode : MetaFieldsMode.values()) {
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " + mode.name() + " --force true");
+ HoodieTableConfig tableConfig = HoodieCLI.getTableMetaClient().getTableConfig();
+ if (tableConfig.getMetaFieldsMode() == mode) {
+ assertEquals(mode.toLegacyPopulateMetaFields(), tableConfig.populateMetaFields(),
+ "populate.meta.fields must be the derived value for mode " + mode);
+ }
+ }
+ }
+
+ @Test
+ public void testSetMetaFieldsModeNoOpWhenAlreadyInTargetMode() {
+ assertTrue(prepareTable());
+ Object first = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(first));
+ // Second call — same target — should be a no-op message.
+ Object second = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(second));
+ assertTrue(second.toString().contains("already in COMMIT_TIME_ONLY"),
+ "expected no-op message, got: " + second);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRejectsUnknownValue() {
+ assertTrue(prepareTable());
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode BOGUS_MODE");
+ // Shell evaluate returns the exception object on failure.
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("BOGUS_MODE"),
+ "expected error message to name the rejected value, got: " + result);
+ }
+
+ @Test
+ public void testSetMetaFieldsModeRefusesOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result));
+ assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"),
+ "expected refusal message, got: " + result);
+
+ // Mode must not have changed.
+ assertEquals(MetaFieldsMode.ALL,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ @Test
+ public void testSetMetaFieldsModeWithForceOnPopulatedTable() throws Exception {
+ assertTrue(prepareTable());
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result));
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ /**
+ * Pins the whole migration matrix rather than spot-checking it. Twenty ordered pairs, each either
+ * a legal narrowing or a refused widening, on a table that already has commits.
+ *
+ * <p>The subtle entries are the mutually-wider siblings: {@code COMMIT_TIME_ONLY} and
+ * {@code FILE_NAME_ONLY} each populate a column the other does not, so neither can migrate to the
+ * other in either direction. A spot-check of "narrowing works, widening does not" would miss that
+ * the relation is a lattice rather than a chain.
+ *
+ * <p>Uses a fresh table per pair so the starting mode can be set without tripping the very guard
+ * under test -- setting the initial mode happens before any commit exists.
+ */
+ @Test
+ public void testSetMetaFieldsModeMigrationMatrixOnPopulatedTable() throws Exception {
+ for (MetaFieldsMode from : MetaFieldsMode.values()) {
+ for (MetaFieldsMode to : MetaFieldsMode.values()) {
+ if (from == to) {
+ continue;
+ }
+ // Fresh table per pair; connect to it so HoodieCLI points at the right one.
+ String pairName = tableName + "_" + from.name() + "_to_" + to.name();
+ String pairPath = tablePath(pairName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + pairPath + " --tableName " + pairName)));
+
+ // Establish the starting mode while the table is still empty, then make it "populated".
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode " + from.name())),
+ "setting the initial mode on an empty table must succeed: " + from);
+ createDummyCommitFileAt(pairPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+ assertEquals(from, HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ boolean widening = to.isWiderThan(from);
+ Object result = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode " + to.name() + " --force true");
+
+ if (widening) {
+ assertFalse(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " adds a meta column and must be refused even with --force");
+ assertTrue(result.toString().contains("widen"),
+ "expected a widening refusal for " + from + " -> " + to + ", got: " + result);
+ assertEquals(from, HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode(),
+ "a refused migration must leave the mode untouched: " + from + " -> " + to);
+ } else {
+ assertTrue(ShellEvaluationResultUtil.isSuccess(result),
+ from + " -> " + to + " drops meta columns and must be allowed with --force, got: " + result);
+ HoodieTableConfig tableConfig = HoodieCLI.getTableMetaClient().getTableConfig();
+ assertEquals(to, tableConfig.getMetaFieldsMode(), from + " -> " + to);
+ assertEquals(to.toLegacyPopulateMetaFields(), tableConfig.populateMetaFields(),
+ "the derived boolean must follow the new mode for " + from + " -> " + to);
+ }
+ }
+ }
+ }
+
+ /** Both mutually-wider directions between the two single-column modes are refused. */
+ @Test
+ public void testSetMetaFieldsModeRefusesBothSiblingDirections() throws Exception {
+ assertTrue(prepareTable());
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY");
+ createDummyCommitFile("20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toSibling = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode FILE_NAME_ONLY --force true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toSibling),
+ "COMMIT_TIME_ONLY -> FILE_NAME_ONLY adds _hoodie_file_name and must be refused");
+ assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+
+ // ...and the reverse, on a table that starts the other way round.
+ String otherName = tableName + "_sibling_reverse";
+ String otherPath = tablePath(otherName);
+ assertTrue(ShellEvaluationResultUtil.isSuccess(
+ shell.evaluate(() -> "create --path " + otherPath + " --tableName " + otherName)));
+ shell.evaluate(() -> "table set-meta-fields-mode --target-mode FILE_NAME_ONLY");
+ createDummyCommitFileAt(otherPath, "20260101000000000");
+ HoodieCLI.refreshTableMetadata();
+
+ Object toOther = shell.evaluate(() ->
+ "table set-meta-fields-mode --target-mode COMMIT_TIME_ONLY --force true");
+ assertFalse(ShellEvaluationResultUtil.isSuccess(toOther),
+ "FILE_NAME_ONLY -> COMMIT_TIME_ONLY adds _hoodie_commit_time and must be refused");
+ assertEquals(MetaFieldsMode.FILE_NAME_ONLY,
+ HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode());
+ }
+
+ private void createDummyCommitFileAt(String tableBasePath, String instantTime) throws IOException {
+ java.nio.file.Path timelineDir =
+ Paths.get(tableBasePath, METAFOLDER_NAME, "timeline");
+ if (!timelineDir.toFile().exists()) {
+ timelineDir.toFile().mkdirs();
+ }
+ String completionTime = instantTime + "1";
+ java.nio.file.Files.createFile(timelineDir.resolve(instantTime + "_" + completionTime + ".commit"));
Review Comment:
🤖 nit: could `createDummyCommitFile` just delegate to `createDummyCommitFileAt(tablePath, instantTime)`? The two methods are identical except for how they arrive at the timeline directory, and `tablePath` is already a field on the test class.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5221947873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed5b5263fa2b6a60abb79ccd696f892230130a7f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222594531
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 109193946a3e70e3b589e252b44ad2eb68092699 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5027030693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470)
* 4f5b3966ea0ed2ec7291ca64b2a364c362e47822 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5029998727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f5b3966ea0ed2ec7291ca64b2a364c362e47822 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222290869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed5b5263fa2b6a60abb79ccd696f892230130a7f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188)
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 109193946a3e70e3b589e252b44ad2eb68092699 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5219405431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f5b3966ea0ed2ec7291ca64b2a364c362e47822 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553)
* 4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5221780935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e13ab243b73fae4568cfb976bec6b577090041f6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185)
* ed5b5263fa2b6a60abb79ccd696f892230130a7f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222323227
**Correcting an earlier note: the tests I said could not run locally do run, and they pass.**
I reported a chain of "pre-existing local build failures" — `hudi-utilities` not compiling on a `KafkaAvroSchemaDeserializer` error, a stale `hudi-utilities-bundle` shadowing `hudi-common`, and `hudi-cli` therefore unable to resolve `MetaFieldsMode.parse`. I attributed all of it to the repo.
It was my build invocation. The root pom defaults `kafka.version` to `2.0.0`, whose `Deserializer` interface lacks the `deserialize(String, Headers, byte[])` overload that `KafkaAvroSchemaDeserializer` declares with `@Override`. The `spark3` profile sets `kafka.version=3.4.1`, which has it. Everything builds with `-Pspark3` — and the bundle was stale precisely *because* rebuilding it hit that same error.
### What now actually runs
| Suite | Result |
|---|---|
| `TestTableCommand` (#19206) | **19/21** — all 13 meta-fields-mode tests pass, including the 20-pair migration matrix and the sibling-direction test |
| `TestHoodieStreamerMetaFieldsMode` (#19205) | **6/6** — including the restart test and the restart-stating-the-legacy-boolean rejection |
Both of those had been committed with a note saying they were compile- and checkstyle-verified only. They are now genuinely executed. The streamer restart rejection was the one I was least comfortable shipping unverified, since it is the breaking change in the release notes.
### The two remaining `TestTableCommand` failures are genuinely pre-existing
`testDefaultCreate` and `testFetchTableSchema` assert an archive path of `.hoodie/history` where the code produces `.hoodie/timeline/history` — a timeline-layout change. The assertion is on master's own copy of the file, last touched by unrelated commit `5124ad76b17d`, and neither PR edits that test. Confirmed by inspection rather than assumed.
Apologies for the noise — the earlier claim was wrong, and it would have led anyone reproducing locally down the same dead end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222983293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 109193946a3e70e3b589e252b44ad2eb68092699 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191)
* ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19206:
URL: https://github.com/apache/hudi/pull/19206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3686863952
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1522,6 +1546,28 @@ public Properties build() {
if (null != populateMetaFields) {
tableConfig.setValue(HoodieTableConfig.POPULATE_META_FIELDS, Boolean.toString(populateMetaFields));
}
+ // Persist the mode in one place. Rules:
+ // - Explicit selective mode wins → validate compatibility with populateMetaFields and write.
+ // - populateMetaFields=false + no explicit mode → NONE (leave property empty for backward compat).
+ // - populateMetaFields=true + no explicit mode → ALL (implicit; leave property empty).
+ if (null != metaFieldsMode) {
Review Comment:
🤖 If a selective `metaFieldsMode` is set while `populateMetaFields` is still null (e.g. a properties source that has `hoodie.meta.fields.mode` but not `hoodie.populate.meta.fields`, since `fromProperties` only calls `setPopulateMetaFields` when the key is present), neither guard fires: the selective mode gets persisted but `POPULATE_META_FIELDS` is never written (line 1546). On read, `populateMetaFields()` defaults to true, so `getMetaFieldsMode()` resolves back to ALL and the selective mode is silently dropped — leaving hoodie.properties claiming e.g. COMMIT_TIME_ONLY while writers actually run in ALL mode. Should build() default `populateMetaFields=false` (or reject) when a selective mode is set without an explicit populate flag?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java:
##########
@@ -158,10 +159,45 @@ public HoodieRowCreateHandle(HoodieTable table,
* @throws IOException
*/
public void write(InternalRow row) throws IOException {
- if (populateMetaFields) {
- writeRow(row);
- } else {
- writeRowNoMetaFields(row);
+ switch (metaFieldsMode) {
+ case ALL:
+ writeRow(row);
+ break;
+ case NONE:
+ writeRowNoMetaFields(row);
+ break;
+ default:
+ writeRowSelectiveMetaFields(row);
+ break;
+ }
+ }
+
+ /**
+ * Selective meta-field write path: populate only the meta columns opted in via
+ * {@code hoodie.meta.fields.mode} — {@code _hoodie_commit_time} and/or {@code _hoodie_file_name}.
+ * The other meta columns stay null on disk. Record key is never populated in this path, so the
+ * record key is not registered with the write support (bloom filter / RLI hooks are meaningless
+ * without the record-key column).
+ */
+ private void writeRowSelectiveMetaFields(InternalRow row) {
Review Comment:
🤖 nit: the magic `5` here is the number of Hoodie meta columns — could you use `HoodieRecord.HOODIE_META_COLUMNS.size()` (or a named constant) so a future reader doesn't have to count the fields to understand where the number comes from?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileWriterFactory.java:
##########
@@ -57,6 +57,9 @@ protected HoodieFileWriter newParquetFileWriter(
String instantTime, StoragePath path, HoodieConfig config, HoodieSchema schema,
TaskContextSupplier taskContextSupplier) throws IOException {
Review Comment:
🤖 nit: `MetaFieldsMode` is referenced with its full package name here — could you add it to the import block at the top of the file instead? The same pattern is used in `HoodieAvroFileWriterFactory` (line 71).
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-4962204097
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9775897985ad1fe221e957f75e7d549d57bddc7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279)
* 7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #19206: feat(cli): add table set-meta-fields-mode command
URL: https://github.com/apache/hudi/pull/19206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-4898293802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9775897985ad1fe221e957f75e7d549d57bddc7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19206:
URL: https://github.com/apache/hudi/pull/19206#discussion_r3737565291
##########
hudi-common/src/main/java/org/apache/hudi/common/model/MetaFieldsMode.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.Locale;
+
+/**
+ * Which of Hudi's meta columns are physically populated on disk.
+ *
+ * <p>Selective modes exist so that tables that opt out of the default {@code populate.meta.fields=true}
+ * can still keep the two columns that matter for downstream operations without paying for the other
+ * three:
+ *
+ * <ul>
+ * <li>{@code _hoodie_commit_time} — required for incremental queries.</li>
+ * <li>{@code _hoodie_file_name} — useful for file-level pruning / investigation lookups.</li>
+ * </ul>
+ *
+ * <p>The remaining three meta columns ({@code _hoodie_commit_seqno}, {@code _hoodie_record_key},
+ * {@code _hoodie_partition_path}) are all-or-nothing — either populate every meta column ({@link #ALL})
+ * or none of them beyond the two selectable ones. If you need any of the remaining columns, set
+ * {@code hoodie.populate.meta.fields=true}.
+ *
+ * <p>This enum is the single source of truth for meta-column population. The legacy boolean
+ * {@code hoodie.populate.meta.fields} is deprecated and consulted only when
+ * {@code hoodie.meta.fields.mode} is absent, so that tables written before the mode property
+ * existed keep their behavior:
+ *
+ * <ul>
+ * <li>{@code populate.meta.fields=true} (or absent) → {@link #ALL} — today's default.</li>
+ * <li>{@code populate.meta.fields=false} → {@link #NONE}.</li>
+ * </ul>
+ *
+ * <p>On-disk representation: the enum {@link #name()} is persisted in {@code hoodie.properties}
+ * under the property {@code hoodie.meta.fields.mode}.
+ */
+public enum MetaFieldsMode {
+ /**
+ * All five Hudi meta columns are populated — today's default.
+ */
+ ALL(true, true),
+
+ /**
+ * No Hudi meta columns are populated. Incremental queries are unsupported. File-level pruning
+ * that depends on {@code _hoodie_file_name} is unsupported.
+ */
+ NONE(false, false),
+
+ /**
+ * Only {@code _hoodie_commit_time} is populated. Incremental queries remain functional; other
+ * meta columns stay null on disk.
+ */
+ COMMIT_TIME_ONLY(true, false),
+
+ /**
+ * Only {@code _hoodie_file_name} is populated. Useful for file-level lookups and debugging;
+ * incremental queries are unsupported.
+ */
+ FILE_NAME_ONLY(false, true),
+
+ /**
+ * Both {@code _hoodie_commit_time} and {@code _hoodie_file_name} are populated.
+ */
+ COMMIT_TIME_AND_FILE_NAME(true, true);
+
+ private final boolean commitTimePopulated;
+ private final boolean fileNamePopulated;
+
+ MetaFieldsMode(boolean commitTimePopulated, boolean fileNamePopulated) {
+ this.commitTimePopulated = commitTimePopulated;
+ this.fileNamePopulated = fileNamePopulated;
+ }
+
+ public boolean isCommitTimePopulated() {
+ return commitTimePopulated;
+ }
+
+ public boolean isFileNamePopulated() {
+ return fileNamePopulated;
+ }
+
+ /**
+ * @return true when all five meta columns are populated (i.e. this is {@link #ALL}). Selective
+ * modes never populate {@code _hoodie_record_key}, {@code _hoodie_partition_path}, or
+ * {@code _hoodie_commit_seqno}.
+ */
+ public boolean isRecordKeyPopulated() {
+ return this == ALL;
+ }
+
+ /**
+ * @return true for the modes that populate some but not all meta columns, i.e. everything except
+ * {@link #ALL} and {@link #NONE}.
+ *
+ * <p>These are the modes the deprecated {@code hoodie.populate.meta.fields} boolean cannot
+ * express, so they are what callers gate on when a code path only understands all-or-nothing meta
+ * fields — writer engines not yet wired for selective population, table versions that predate the
+ * mode property, and validation that must not let a two-state writer speak for a five-state table.
+ */
+ public boolean isSelective() {
+ return this != ALL && this != NONE;
+ }
+
+ /**
+ * Resolve the effective mode from any {@link HoodieConfig} that may carry the two properties —
+ * a table config, a write config, or a bare config built from write options. Preferred over the
+ * two-argument overload: it keeps the property keys and the precedence rule in one place instead
+ * of repeating them at every call site.
+ */
+ public static MetaFieldsMode resolve(HoodieConfig config) {
+ return resolve(config.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE),
+ config.getBooleanOrDefault(HoodieTableConfig.POPULATE_META_FIELDS));
+ }
+
+ /**
+ * Resolve the effective mode. {@code hoodie.meta.fields.mode} is the source of truth; the
+ * deprecated {@code hoodie.populate.meta.fields} boolean is a fallback for tables written before
+ * the mode property existed. Precedence:
+ *
+ * <ul>
+ * <li>non-empty mode → the parsed enum value (the legacy boolean is not consulted).</li>
+ * <li>null/empty mode + {@code populateMetaFields=false} → {@link #NONE}.</li>
+ * <li>null/empty mode + {@code populateMetaFields=true} → {@link #ALL}.</li>
+ * </ul>
+ *
+ * @param rawMode raw {@code hoodie.meta.fields.mode} value; may be null or empty.
+ * @param legacyPopulateMetaFields value of the deprecated {@code hoodie.populate.meta.fields}.
+ * @throws IllegalArgumentException when the raw mode value does not match any enum value. This
+ * includes the pre-enum comma-separated format — callers that upgrade an old table must
+ * migrate the value through the hudi-cli.
+ */
+ public static MetaFieldsMode resolve(String rawMode, boolean legacyPopulateMetaFields) {
+ if (StringUtils.isNullOrEmpty(rawMode)) {
+ return legacyPopulateMetaFields ? ALL : NONE;
+ }
+ return parse(rawMode);
+ }
+
+ /**
+ * Parse a raw {@code hoodie.meta.fields.mode} value into an enum constant, with a message that
+ * lists the allowed values. Prefer this over {@link #valueOf(String)} for user-supplied input.
+ */
+ public static MetaFieldsMode parse(String rawMode) {
+ try {
+ // Case-insensitive: users hand-editing hoodie.properties or passing write options should not
+ // have to match the enum's casing exactly.
+ return MetaFieldsMode.valueOf(rawMode.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String.format(
+ "Unsupported value '%s' for hoodie.meta.fields.mode. Allowed values: %s, %s, %s, %s, %s.",
+ rawMode, ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME), e);
Review Comment:
🤖 nit: the allowed-values list in the error message is hardcoded, so adding a sixth enum constant would silently leave the message stale. Could you replace it with `Arrays.stream(values()).map(Enum::name).collect(Collectors.joining(", "))` so it stays in sync automatically?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,13 +1545,80 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ /**
+ * Adopt the table's {@code hoodie.meta.fields.mode} when this writer did not state one.
+ *
+ * <p>The mode is a table property: it is settable at table creation, through hudi-cli, or by an
+ * upgrade, and never by an ordinary write. Callers routinely build a write config without
+ * restating the table's meta-field settings — table services do, and so does a restarted
+ * StreamSync — and such a writer must write what the table already advertises rather than silently
+ * narrowing it. Without this, an unstated writer resolves to {@code NONE} (via the deprecated
+ * {@code hoodie.populate.meta.fields} fallback) and writes null meta columns into a table whose
+ * earlier files have them populated.
+ *
+ * <p>Inheritance applies only when the writer states <em>neither</em> property. If it explicitly
+ * set the mode or the deprecated boolean, that value is left alone so the comparison below can
+ * reject it on a mismatch: a user who deliberately passed {@code populate.meta.fields=false}
+ * against an {@code ALL} table should be told the setting conflicts, not have it silently
+ * overridden.
+ *
+ * @return true when the writer stated neither property, i.e. the mode was inherited and cannot
+ * disagree with the table.
+ */
+ private static boolean inferMetaFieldsModeFromTable(HoodieTableConfig tableConfig, HoodieWriteConfig writeConfig) {
Review Comment:
🤖 `inferMetaFieldsModeFromTable` mutates `writeConfig` as a side effect, and that mutation is what makes selective/NONE writers emit the right meta columns — the create/merge handles resolve the mode from this same config later during the write. It works today because `initTable` validates the shared `config` in place before any handle is built, but it's a fairly hidden dependency for something that silently drives on-disk layout: a caller that validates a defensive copy, or reads the mode before `validateAgainstTableProperties` runs, would quietly write the wrong meta columns with no error. Would it be worth lifting the inheritance into an explicit config-normalization step (or renaming to signal it mutates), so the data-correctness dependency isn't buried inside a `validate` method? @yihua
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,13 +1545,80 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ /**
+ * Adopt the table's {@code hoodie.meta.fields.mode} when this writer did not state one.
+ *
+ * <p>The mode is a table property: it is settable at table creation, through hudi-cli, or by an
+ * upgrade, and never by an ordinary write. Callers routinely build a write config without
+ * restating the table's meta-field settings — table services do, and so does a restarted
+ * StreamSync — and such a writer must write what the table already advertises rather than silently
+ * narrowing it. Without this, an unstated writer resolves to {@code NONE} (via the deprecated
+ * {@code hoodie.populate.meta.fields} fallback) and writes null meta columns into a table whose
+ * earlier files have them populated.
+ *
+ * <p>Inheritance applies only when the writer states <em>neither</em> property. If it explicitly
+ * set the mode or the deprecated boolean, that value is left alone so the comparison below can
+ * reject it on a mismatch: a user who deliberately passed {@code populate.meta.fields=false}
+ * against an {@code ALL} table should be told the setting conflicts, not have it silently
+ * overridden.
+ *
+ * @return true when the writer stated neither property, i.e. the mode was inherited and cannot
+ * disagree with the table.
+ */
+ private static boolean inferMetaFieldsModeFromTable(HoodieTableConfig tableConfig, HoodieWriteConfig writeConfig) {
+ boolean statedMode = writeConfig.contains(HoodieTableConfig.META_FIELDS_MODE)
Review Comment:
🤖 nit: `inferMetaFieldsModeFromTable` is declared to return `boolean`, but its only call site discards the value — have you considered changing the return type to `void`? The Javadoc for the return value (`@return true when the writer stated neither property`) is meaningful, but since no caller acts on it, a reader will search for where it matters and find nothing.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5222042078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed5b5263fa2b6a60abb79ccd696f892230130a7f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188)
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5221156632
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179)
* e13ab243b73fae4568cfb976bec6b577090041f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5223433017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(cli): add table set-meta-fields-mode command [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19206:
URL: https://github.com/apache/hudi/pull/19206#issuecomment-5338241357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15279",
"triggerID" : "c9775897985ad1fe221e957f75e7d549d57bddc7",
"triggerType" : "PUSH"
}, {
"hash" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15470",
"triggerID" : "7112d91f9e11d5521cb8b23a6ae2cc39fff1b55c",
"triggerType" : "PUSH"
}, {
"hash" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15553",
"triggerID" : "4f5b3966ea0ed2ec7291ca64b2a364c362e47822",
"triggerType" : "PUSH"
}, {
"hash" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16179",
"triggerID" : "4f56c57b7c06c9832ff1b4d2a57377fab1f0e2c0",
"triggerType" : "PUSH"
}, {
"hash" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16185",
"triggerID" : "e13ab243b73fae4568cfb976bec6b577090041f6",
"triggerType" : "PUSH"
}, {
"hash" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16188",
"triggerID" : "ed5b5263fa2b6a60abb79ccd696f892230130a7f",
"triggerType" : "PUSH"
}, {
"hash" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a55c0ffd148c1b02093a1867fdc81872bab810c5",
"triggerType" : "PUSH"
}, {
"hash" : "109193946a3e70e3b589e252b44ad2eb68092699",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16191",
"triggerID" : "109193946a3e70e3b589e252b44ad2eb68092699",
"triggerType" : "PUSH"
}, {
"hash" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16195",
"triggerID" : "ddf4afb2d9d0ff02c7a7544c8f79dd21ecb628c1",
"triggerType" : "PUSH"
}, {
"hash" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16255",
"triggerID" : "097c8d95b3abfc325314c7cc053e113ec014f725",
"triggerType" : "PUSH"
}, {
"hash" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16376",
"triggerID" : "0f15545398cbb26bbaa6ef652e67e0778bb9f366",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a55c0ffd148c1b02093a1867fdc81872bab810c5 UNKNOWN
* 0f15545398cbb26bbaa6ef652e67e0778bb9f366 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16376)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org