You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "claudevdm (via GitHub)" <gi...@apache.org> on 2026/09/18 03:42:42 UTC
[PR] AddFiles: Add dry_run [beam]
claudevdm opened a new pull request, #40167:
URL: https://github.com/apache/beam/pull/40167
AddFiles: `dry_run` reports what the schema pre-pass would do
`SchemaEvolutionConfig.setDryRun(true)` (`dry_run: true` in YAML) turns AddFiles into a report-only transform. The read side runs as usual (footers, distinct schemas); instead of committing and registering, a new `dry_run_report` output gets one row per distinct file schema plus a `create` row, marker rows for unreadable and ORC/Avro files, and a summary row. `snapshots` and `errors` are empty. The output only exists when `dry_run` is set, so existing YAML pipelines are unaffected.
The diff has two halves. Reading them in this order is easiest.
**1. `CommitSchemaUnion`: the commit now plans before it writes** (about a third of the diff, plus `SchemaDelta` and `CommitSchemaUnionTest`).
Everything a commit decides is pulled into `Plan`, built by `plan()`: load the table (or note it is missing), classify and fold on scratch transactions (or fold for creation), then the configuration checks against the schema the run would end with. `commitOnce` and `create` consume the plan; `DryRunReport` turns the same plan into rows. A check added to the plan reaches both, so the report cannot drift from the commit. In particular a schema that is fine against the table but conflicts with another schema of the input is reported with the blame a real run assigns.
**2. The dry run itself**: `DryRunReport` (new), the `expand` wiring in `AddFiles`, the `dry_run` knob in `SchemaEvolutionConfig` and the YAML provider, the marker keys in `ReadFooterSchema`, the transform-level tests in `AddFilesTest`, and a YAML fixture that runs a dry run against a missing table, writes the report to JSON and asserts the full rows.
### Report rows
| column | meaning |
|---|---|
| `schema_key` | short murmur3 key of the schema JSON; `create`, `unread`, `unchecked`, `summary` for the other rows |
| `schema` | canonical schema JSON; the union the table would be created with on the `create` row |
| `num_files` | files carrying it |
| `changes` | the `SchemaDelta` descriptions; one `create optional <name> <type>` per column on the `create` row; the totals line plus any table-level change (name mapping) on the summary |
| `allowed` | whether a real run would accept it |
| `reason` | why not, else empty; the consequence on the summary |
| `would_create_table` | false whenever a real run would not create the table, including when it would fail first |
| `summary` | true on the summary row |
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40167:
URL: https://github.com/apache/beam/pull/40167#issuecomment-5725316818
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40167?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 58.47%. Comparing base ([`d375603`](https://app.codecov.io/gh/apache/beam/commit/d37560364756bface6fcffad8575e6c47cb91d5e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f436398`](https://app.codecov.io/gh/apache/beam/commit/f4363986ce64cab14b312cd71622441bb7566271?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #40167 +/- ##
=========================================
Coverage 58.47% 58.47%
Complexity 13593 13593
=========================================
Files 2576 2576
Lines 269266 269266
Branches 11079 11079
=========================================
Hits 157441 157441
+ Misses 105863 105862 -1
- Partials 5962 5963 +1
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/40167/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [java](https://app.codecov.io/gh/apache/beam/pull/40167/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.70% <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40167?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40167:
URL: https://github.com/apache/beam/pull/40167#issuecomment-5724840196
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #40167:
URL: https://github.com/apache/beam/pull/40167#issuecomment-5724833825
R: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #40167:
URL: https://github.com/apache/beam/pull/40167#discussion_r4066824317
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/CommitSchemaUnion.java:
##########
@@ -153,6 +179,33 @@ private static String truncate(String json) {
+ " chars truncated)";
}
+ /** Where each distinct file schema of the window stands while the plan is worked out. */
+ private static final class Verdicts {
+ final List<SchemaToMerge> toMerge = new ArrayList<>();
+ final List<IncompatibleSchema> incompatible = new ArrayList<>();
+
+ void refuse(CollectDistinctSchemas.SchemaGroup group, String reason) {
+ incompatible.add(new IncompatibleSchema(group.getSchemaJson(), group.getFiles(), reason));
+ }
+
+ void refuse(Conflict conflict) {
+ toMerge.remove(conflict.schema);
+ incompatible.add(
+ new IncompatibleSchema(conflict.schema.json, conflict.schema.files, conflict.reason));
+ }
+ }
+
+ /** A schema that passed classification but cannot be staged after the ones before it. */
+ private static final class Conflict {
Review Comment:
> but cannot be staged after the ones before it.
Can you clarify what this means?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/CommitSchemaUnion.java:
##########
@@ -191,98 +255,293 @@ static long commit(
Thread.currentThread().interrupt();
throw e;
}
- LOG.info(
- "Schema commit attempt {}/{} for {} failed; reloading and rebuilding",
- attempt,
- MAX_ATTEMPTS,
- tableId,
- e);
}
}
}
- private static long commitOnce(
+ /**
+ * What one schema commit would do, as data: which distinct file schemas it merges into the table,
+ * which it refuses and why, and the schema the table ends up with. Computed on scratch
+ * transactions; the commit executes it and the dry run reports it. Adding a check here is the
+ * only way to add one, so the two cannot drift.
+ */
+ abstract static class Plan {
Review Comment:
nit: Worth putting all these `Plan` related classes and logic in a separate file
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/CommitSchemaUnion.java:
##########
@@ -191,98 +255,293 @@ static long commit(
Thread.currentThread().interrupt();
throw e;
}
- LOG.info(
- "Schema commit attempt {}/{} for {} failed; reloading and rebuilding",
- attempt,
- MAX_ATTEMPTS,
- tableId,
- e);
}
}
}
- private static long commitOnce(
+ /**
+ * What one schema commit would do, as data: which distinct file schemas it merges into the table,
+ * which it refuses and why, and the schema the table ends up with. Computed on scratch
+ * transactions; the commit executes it and the dry run reports it. Adding a check here is the
+ * only way to add one, so the two cannot drift.
+ */
+ abstract static class Plan {
+ final List<SchemaToMerge> schemasToMerge;
+ final List<IncompatibleSchema> incompatibleSchemas;
+
+ /** The union to replay, or the schema to create the table with; null when nothing changes. */
+ final @Nullable Schema newSchema;
+
+ /** Problems the configuration raises against the planned schema, as messages. */
+ final List<String> configProblems;
+
+ private Plan(Verdicts verdicts, @Nullable Schema newSchema, List<String> configProblems) {
+ this.schemasToMerge = verdicts.toMerge;
+ this.incompatibleSchemas = verdicts.incompatible;
+ this.newSchema = newSchema;
+ this.configProblems = configProblems;
+ }
+
+ /** Null when the schema is incompatible, or when the existing table already covers it. */
+ @Nullable SchemaToMerge toMerge(String schemaJson) {
+ for (SchemaToMerge item : schemasToMerge) {
+ if (item.json.equals(schemaJson)) {
+ return item;
+ }
+ }
+ return null;
+ }
+
+ /** Null when the schema is not incompatible. */
+ @Nullable String incompatibleReason(String schemaJson) {
Review Comment:
javadoc seems contradictory to the method?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +145,25 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output. Its rows are told apart by row_type: schema"
+ + " (one per distinct file schema: files, changes a real run would make, whether they"
+ + " are allowed and why not), create (the table a real run would create), unreadable,"
Review Comment:
Extremely long and convoluted for a public facing description. Can you please trim it down and simplify for readability?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DryRunReport.java:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.metrics.Metrics.counter;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import org.apache.beam.sdk.io.iceberg.SchemaEvolutionConfig.IncompatibleSchemaHandling;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.hash.Hashing;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Dry run of the schema pre-pass: computes the {@link CommitSchemaUnion#plan} a real commit would,
+ * on scratch transactions only, and reports it without committing or registering anything. Planning
+ * the real fold rather than classifying each schema alone is what makes a schema that is fine
+ * against the table but conflicts with another schema of the input come out with the blame a real
+ * run would assign.
+ *
+ * <p>Rows, by {@code row_type}: {@code schema}, one per distinct schema, with the changes a real
+ * run would make on an existing table ({@code schema_key} is a short hash to group by); {@code
+ * create} for the table a real run would create, once, from the union of the allowed schemas;
+ * {@code unreadable} for files whose schema could not be read; {@code unchecked} for ORC and Avro
+ * files the per-file checks cannot verify; and {@code summary}. The summary is {@code allowed} when
+ * no schema is incompatible and the configuration raises no problem; its {@code changes} hold the
+ * totals line and any table-level change a real run would make without a schema change, such as
+ * regenerating the name mapping. {@code would_create_table} is false whenever a real run would not
+ * create the table, including when it would fail first.
+ *
+ * <p>The counters and the totals count checked Parquet files by their schema's verdict; unchecked
+ * files count separately even when ACCEPT registers them, so the unchecked row can be {@code
+ * allowed} while its files are outside {@code numDryRunFilesAllowed}. Pin evidence is per file (the
+ * footer's null counts), so a pin violation or an unproven pin is not predicted here.
+ */
+class DryRunReport extends DoFn<List<CollectDistinctSchemas.SchemaGroup>, Row> {
+ private static final Logger LOG = LoggerFactory.getLogger(DryRunReport.class);
+
+ static final String FILES_ALLOWED_COUNTER = "numDryRunFilesAllowed";
+ static final String FILES_INCOMPATIBLE_COUNTER = "numDryRunFilesIncompatible";
+ static final String FILES_UNREADABLE_COUNTER = "numDryRunFilesUnreadable";
+ static final String FILES_UNCHECKED_COUNTER = "numDryRunFilesUnchecked";
+ static final String CONFIG_PROBLEMS_COUNTER = "numDryRunConfigProblems";
+
+ private static final Counter numFilesAllowed = counter(DryRunReport.class, FILES_ALLOWED_COUNTER);
+ private static final Counter numFilesIncompatible =
+ counter(DryRunReport.class, FILES_INCOMPATIBLE_COUNTER);
+ private static final Counter numFilesUnreadable =
+ counter(DryRunReport.class, FILES_UNREADABLE_COUNTER);
+ private static final Counter numFilesUnchecked =
+ counter(DryRunReport.class, FILES_UNCHECKED_COUNTER);
+ private static final Counter numConfigProblems =
+ counter(DryRunReport.class, CONFIG_PROBLEMS_COUNTER);
+
+ static final Schema REPORT_SCHEMA =
+ Schema.builder()
+ .addStringField("row_type")
+ .addStringField("schema_key")
+ .addStringField("schema")
+ .addInt64Field("num_files")
+ .addArrayField("changes", Schema.FieldType.STRING)
+ .addBooleanField("allowed")
+ .addStringField("reason")
+ .addBooleanField("would_create_table")
+ .build();
+
+ static final String SCHEMA_ROW = "schema";
+ static final String CREATE_ROW = "create";
+ static final String UNREADABLE_ROW = "unreadable";
+ static final String UNCHECKED_ROW = "unchecked";
+ static final String SUMMARY_ROW = "summary";
+
+ static final String NAME_MAPPING_CHANGE =
+ "regenerate the name mapping property to cover the schema";
+
+ /** Wide inputs would otherwise put every column of every schema into one log entry. */
+ private static final int MAX_RENDERED_ROWS = 50;
+
+ private static final int MAX_RENDERED_CHANGES = 10;
+
+ static final String UNREADABLE_REASON =
+ "the file schema could not be read (unknown format, or an unreadable footer);"
+ + " a real run routes these files to the error output";
+
+ static final String UNCHECKED_REJECTED_REASON =
+ "ORC and Avro files cannot be checked; a real run routes these files to the error output"
+ + " (UnverifiableFileHandling.REJECT)";
+
+ static final String UNCHECKED_ACCEPTED_REASON =
+ "ORC and Avro files cannot be checked; a real run registers these files unchecked"
+ + " (UnverifiableFileHandling.ACCEPT)";
+
+ private final IcebergCatalogConfig catalogConfig;
+ private final String identifier;
+ private final CommitSchemaUnion.Settings settings;
+ private transient @MonotonicNonNull Catalog catalog;
+
+ DryRunReport(
+ IcebergCatalogConfig catalogConfig, String identifier, CommitSchemaUnion.Settings settings) {
+ this.catalogConfig = catalogConfig;
+ this.identifier = identifier;
+ this.settings = settings;
+ }
+
+ /** One report row before it is a Row: the summary and the rendered log read these. */
+ private static final class Line {
+ final String rowType;
+ final String schemaKey;
+ final String schema;
+ final long files;
+ final List<String> changes;
+ final boolean allowed;
+ final String reason;
+
+ Line(
+ String rowType,
+ String schemaKey,
+ String schema,
+ long files,
+ List<String> changes,
+ boolean allowed,
+ String reason) {
+ this.rowType = rowType;
+ this.schemaKey = schemaKey;
+ this.schema = schema;
+ this.files = files;
+ this.changes = changes;
+ this.allowed = allowed;
+ this.reason = reason;
+ }
+
+ Row toRow(boolean wouldCreateTable) {
+ return Row.withSchema(REPORT_SCHEMA)
+ .withFieldValue("row_type", rowType)
+ .withFieldValue("schema_key", schemaKey)
+ .withFieldValue("schema", schema)
+ .withFieldValue("num_files", files)
+ .withFieldValue("changes", changes)
+ .withFieldValue("allowed", allowed)
+ .withFieldValue("reason", reason)
+ .withFieldValue("would_create_table", wouldCreateTable)
+ .build();
+ }
+ }
+
+ /** The window's schema groups: the ones the plan sees, and the files that contribute none. */
+ private static final class Input {
+ final List<CollectDistinctSchemas.SchemaGroup> readable = new ArrayList<>();
+ long unreadableFiles;
+ long uncheckedFiles;
+
+ static Input of(List<CollectDistinctSchemas.SchemaGroup> schemas) {
+ Input input = new Input();
+ for (CollectDistinctSchemas.SchemaGroup group : schemas) {
+ if (group.getSchemaJson().equals(ReadFooterSchema.UNREADABLE_KEY)) {
+ input.unreadableFiles += group.getFiles();
+ } else if (group.getSchemaJson().equals(ReadFooterSchema.UNCHECKED_FORMAT_KEY)) {
+ input.uncheckedFiles += group.getFiles();
+ } else {
+ input.readable.add(group);
+ }
+ }
+ return input;
+ }
+ }
+
+ private static final class Totals {
+ int allowedSchemas;
+ long allowedFiles;
+ int incompatibleSchemas;
+ long incompatibleFiles;
+
+ static Totals of(List<Line> schemaLines) {
+ Totals totals = new Totals();
+ for (Line line : schemaLines) {
+ if (line.allowed) {
+ totals.allowedSchemas++;
+ totals.allowedFiles += line.files;
+ } else {
+ totals.incompatibleSchemas++;
+ totals.incompatibleFiles += line.files;
+ }
+ }
+ return totals;
+ }
+ }
+
+ @ProcessElement
+ public void process(
+ @Element List<CollectDistinctSchemas.SchemaGroup> schemas, OutputReceiver<Row> out) {
+ if (catalog == null) {
+ catalog = catalogConfig.catalog();
+ }
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(identifier);
+ Input input = Input.of(schemas);
+ CommitSchemaUnion.Plan plan =
+ CommitSchemaUnion.plan(catalog, tableId, input.readable, settings);
+
+ List<Line> lines = new ArrayList<>();
+ for (CollectDistinctSchemas.SchemaGroup group : input.readable) {
+ lines.add(schemaLine(group, plan));
+ }
+ Totals totals = Totals.of(lines);
+
+ boolean repairsNameMapping =
+ plan instanceof CommitSchemaUnion.EvolutionPlan
+ && ((CommitSchemaUnion.EvolutionPlan) plan).repairsNameMapping;
+ CommitSchemaUnion.@Nullable CreationPlan creation = null;
+ if (plan instanceof CommitSchemaUnion.CreationPlan) {
+ creation = (CommitSchemaUnion.CreationPlan) plan;
+ }
+ @Nullable String creationProblem = creation == null ? null : creation.problem;
+ boolean wouldFail =
+ creationProblem != null
+ || (settings.handling == IncompatibleSchemaHandling.FAIL_PIPELINE
+ && (totals.incompatibleSchemas > 0 || !plan.configProblems.isEmpty()));
+ boolean wouldCreateTable = creation != null && creation.canCreate() && !wouldFail;
+
+ if (creation != null && creation.newSchema != null) {
+ lines.add(
+ createLine(creation.newSchema, totals.allowedFiles, wouldCreateTable, creationProblem));
+ }
+ if (input.unreadableFiles > 0) {
+ lines.add(unreadableLine(input.unreadableFiles));
+ }
+ if (input.uncheckedFiles > 0) {
+ lines.add(uncheckedLine(input.uncheckedFiles));
+ }
Review Comment:
Would it be better to have one single output row that contains the full report?
It would let the user expect only one output instead of potentially many different outputs.
Also not all columns are relevant for the different outputs we currently have.
Haven't thought too deeply about this though, there could be downsides and increased complexity to putting everything in one output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm merged PR #40167:
URL: https://github.com/apache/beam/pull/40167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AddFiles: Add dry_run [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #40167:
URL: https://github.com/apache/beam/pull/40167#discussion_r4075955339
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +146,19 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output with one row that describes what a real run"
Review Comment:
```suggestion
+ " and emits a `dry_run_report` output with one row that describes what a real run"
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +146,19 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output with one row that describes what a real run"
+ + " would do. Its allowed field is true when every file schema can be merged and the"
+ + " configuration raises no problem; otherwise its reason field says what a real run"
+ + " would do about it (fail, or route the files to the error output). Its schemas"
Review Comment:
```suggestion
+ " would do about it (fail, or route the files to the error output). Its `schemas`"
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +146,19 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output with one row that describes what a real run"
+ + " would do. Its allowed field is true when every file schema can be merged and the"
Review Comment:
```suggestion
+ " would do. Its `allowed` field is true when every file schema can be merged and the"
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +146,19 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output with one row that describes what a real run"
+ + " would do. Its allowed field is true when every file schema can be merged and the"
+ + " configuration raises no problem; otherwise its reason field says what a real run"
Review Comment:
```suggestion
+ " configuration raises no problem; otherwise its `reason` field says what a real run"
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java:
##########
@@ -145,6 +146,19 @@ public static Builder builder() {
+ " Requires schema_evolution_options.")
public abstract @Nullable List<String> getRequiredColumns();
+ @SchemaFieldDescription(
+ "When true, nothing is committed or registered: the transform reads the files' schemas"
+ + " and emits a dry_run_report output with one row that describes what a real run"
+ + " would do. Its allowed field is true when every file schema can be merged and the"
+ + " configuration raises no problem; otherwise its reason field says what a real run"
+ + " would do about it (fail, or route the files to the error output). Its schemas"
+ + " field lists each distinct file schema with the changes a real run would make for"
+ + " it and, when it cannot be merged, why. The output only exists when this is set;"
+ + " consume it as input: <this transform's name>.dry_run_report. Against a missing"
Review Comment:
```suggestion
+ " consume it as input: `<this transform's name>.dry_run_report`. Against a missing"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org