You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jrmccluskey (via GitHub)" <gi...@apache.org> on 2026/08/25 15:04:32 UTC
[PR] Implement Iceberg Table Metadata Driver transform [beam]
jrmccluskey opened a new pull request, #39883:
URL: https://github.com/apache/beam/pull/39883
Implements the Iceberg table metadata query transform and materializes the table metadata to be used downstream as a side input.
Part of #39723
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3915882786
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
Review Comment:
I'm not sure I'm following the ask here. Would we want to emit batches of table metadata downstream in that context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3918584510
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,723 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ private static final TableIdentifier TABLE_ID = TableIdentifier.of("default", "table");
+
+ private static final DynamicDestinations SINGLE_TABLE_DYNAMIC_DESTINATIONS =
+ DynamicDestinations.singleTable(TABLE_ID, BEAM_SCHEMA);
+
+ private static final DynamicDestinations DYNAMIC_DESTINATIONS =
+ new DynamicDestinations() {
+ @Override
+ public Schema getDataSchema() {
+ return BEAM_SCHEMA;
+ }
+
+ @Override
+ public Row getData(Row element) {
+ return element;
+ }
+
+ @Override
+ public IcebergDestination instantiateDestination(String destination) {
+ return IcebergDestination.builder()
+ .setTableIdentifier(IcebergUtils.parseTableIdentifier(destination))
+ .build();
+ }
+
+ @Override
+ public String getTableStringIdentifier(ValueInSingleWindow<Row> element) {
+ return element.getValue().getString("dest");
+ }
+ };
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ Table realTable = getCatalog().createTable(TABLE_ID, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(SINGLE_TABLE_DYNAMIC_DESTINATIONS)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(TABLE_ID);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ List<Row> rows =
+ ImmutableList.of(
+ Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.table_b").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.table_c").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(4L, "v4", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(5L, "v5", "default.table_b").build());
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(3, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.table_a"));
+ assertTrue(map.containsKey("default.table_b"));
+ assertTrue(map.containsKey("default.table_c"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWindowedDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 100; i++) {
+ String dest = (i % 2 == 0) ? "default.t1" : "default.t2";
+ rows.add(Row.withSchema(BEAM_SCHEMA).addValues((long) i, "val_" + i, dest).build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.t1"));
+ assertTrue(map.containsKey("default.t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testUnboundedGlobalWindowStreamingDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "stream_t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "stream_t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.stream_t1").build();
+ Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.stream_t2").build();
+ Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.stream_t1").build();
+
+ TestStream<Row> stream =
+ TestStream.create(RowCoder.of(BEAM_SCHEMA))
+ .advanceWatermarkTo(new Instant(0))
+ .addElements(row1)
+ .addElements(row2)
+ .addElements(row3)
+ .advanceProcessingTime(Duration.standardSeconds(5))
+ .advanceWatermarkToInfinity();
+
+ PCollection<Row> input = pipeline.apply("StreamInput", stream);
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .setRefreshInterval(Duration.standardSeconds(2))
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.stream_t1"));
+ assertTrue(map.containsKey("default.stream_t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMetadataRefreshedAcrossIntervals() {
Review Comment:
This test looks good, but can we please add another similar one consuming it specifically as a side input?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5413699199
Assigning reviewers:
R: @chamikaramj for label java.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3909078154
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
Review Comment:
How are we going to control parallelism? Ideally we'd pass tableIds to only a few tables* so they can do sequential loadTable calls.
*this can be configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3925226501
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
Review Comment:
Reshuffle via random key with a fixed number of buckets would do that, I'll see how that looks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5547856969
Resolution on the notes + some cleanup from an agentic review pass @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3915468846
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Looking at Deduplicate it's a stateful DoFn, which the initial problem statement and design scope seemed insistent on avoiding (mainly for the potential bottleneck from shuffles.) Is there a particular reason it's preferred for streaming workloads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3917153628
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Yeah the more I thought about it the less I was worried about the bottleneck in a streaming context since we're 1) passing pretty lightweight objects and 2) anticipate relatively small bundle sizes and worker counts for streaming workloads. If it was the batch patch I would be more concerned since that's the use-case where the potential to DDOS during queries is high
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5548183214
Let's make sure tests pass too
```
2026-09-05T00:14:40.2545718Z TableMetadataDriverTest > testStreamingMixedExistingAndNonExistentTables FAILED
2026-09-05T00:14:40.2547824Z java.lang.AssertionError
2026-09-05T00:14:40.2549118Z at org.junit.Assert.fail(Assert.java:87)
2026-09-05T00:14:40.2551215Z at org.junit.Assert.assertTrue(Assert.java:42)
2026-09-05T00:14:40.2552529Z at org.junit.Assert.assertNotNull(Assert.java:713)
2026-09-05T00:14:40.2553834Z at org.junit.Assert.assertNotNull(Assert.java:723)
2026-09-05T00:14:40.2555910Z at org.apache.beam.sdk.io.iceberg.TableMetadataDriverTest$9.processElement(TableMetadataDriverTest.java:676)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3867434851
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, samples up to a maximum number of tables, loads their declarative metadata from the
+ * Iceberg catalog, and emits {@link KV} pairs of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. If the number of distinct tables in a window
+ * exceeds {@code maxTables}, up to {@code maxTables} tables are sampled into the broadcasted view,
+ * while remaining destinations can fall back to worker-local catalog loading.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final int DEFAULT_MAX_TABLES = 100;
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract int getMaxTables();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder().setMaxTables(DEFAULT_MAX_TABLES);
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaxTables(int maxTables);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Preconditions.checkArgument(
+ driver.getMaxTables() > 0,
+ "maxTables must be greater than 0, got %s",
+ driver.getMaxTables());
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates a {@link PCollectionView} of {@link
+ * Map} of table identifier strings to {@link SerializableTableSpec} using {@link
+ * #DEFAULT_MAX_TABLES}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, DEFAULT_MAX_TABLES);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with a custom {@code maxTables} limit and
+ * creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maxTables maximum distinct tables to poll and broadcast per window.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ int maxTables) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaxTables(maxTables)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ PCollection<String> distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> sampledTableIds =
+ distinctTableIds.apply("SampleTableIds", Sample.any(getMaxTables()));
+
+ return sampledTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn extends DoFn<String, KV<String, SerializableTableSpec>> {
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString, OutputReceiver<KV<String, SerializableTableSpec>> out) {
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ Table table = catalogConfig.catalog().loadTable(tableId);
Review Comment:
Note that a bad table string here will cause the whole bundle to fail. Is this intended or should we skip bad records and try other elements ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3895514758
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, samples up to a maximum number of tables, loads their declarative metadata from the
+ * Iceberg catalog, and emits {@link KV} pairs of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. If the number of distinct tables in a window
+ * exceeds {@code maxTables}, up to {@code maxTables} tables are sampled into the broadcasted view,
+ * while remaining destinations can fall back to worker-local catalog loading.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final int DEFAULT_MAX_TABLES = 100;
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract int getMaxTables();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder().setMaxTables(DEFAULT_MAX_TABLES);
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaxTables(int maxTables);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Preconditions.checkArgument(
+ driver.getMaxTables() > 0,
+ "maxTables must be greater than 0, got %s",
+ driver.getMaxTables());
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates a {@link PCollectionView} of {@link
+ * Map} of table identifier strings to {@link SerializableTableSpec} using {@link
+ * #DEFAULT_MAX_TABLES}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, DEFAULT_MAX_TABLES);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with a custom {@code maxTables} limit and
+ * creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maxTables maximum distinct tables to poll and broadcast per window.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ int maxTables) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaxTables(maxTables)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ PCollection<String> distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Probably worth deduplicating a different way then, don't want to add an extra restriction here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3908840145
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Deduplicate transform is generally a better alternative for streaming mode. it also has a `.withDuration(refreshInterval)` method that aligns well here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3908752454
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,834 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ TableIdentifier tableId = TableIdentifier.of("default", "single_table");
+ Table realTable = getCatalog().createTable(tableId, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations = DynamicDestinations.singleTable(tableId, BEAM_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(tableId);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations =
Review Comment:
This dynamicDestinations object seems to be repeated across multiple tests. Can we create it outside and share it among tests? Would help with readability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5588868953
Weird that both that unit test + spotless were passing locally but had issues on CI (the unit test specifically seems to have been a flake that I have been unable to reproduce at the moment.) I did get things cleaned up to address the nits + fix the spotless issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3915945298
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Well I feel silly, adding a schema evolution test case actually found that Distinct didn't re-emit the pane in an unbounded streaming context. That explains it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3917002721
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
Review Comment:
Sorry lemme clarify. I think as it stands, each tableId (after deduplication) can go to a separate worker. If we're writing to many tables, we can end up with many concurrent loadTable calls (one for each table).
Was wondering if it makes sense to reshuffle these table string outputs to a fixed N workers so that we only have at most N concurrent loadTable calls. If there's many tables, the workers can call loadTable on each one sequentially
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3925131289
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Yeah that shouldn't impact coders or graph shape, we can play with that later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3867228224
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ TableIdentifier tableId = TableIdentifier.of("default", "single_table");
+ Table realTable = getCatalog().createTable(tableId, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations = DynamicDestinations.singleTable(tableId, BEAM_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(tableId);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations =
+ new DynamicDestinations() {
+ @Override
+ public Schema getDataSchema() {
+ return BEAM_SCHEMA;
+ }
+
+ @Override
+ public Row getData(Row element) {
+ return element;
+ }
+
+ @Override
+ public IcebergDestination instantiateDestination(String destination) {
+ return IcebergDestination.builder()
+ .setTableIdentifier(IcebergUtils.parseTableIdentifier(destination))
+ .build();
+ }
+
+ @Override
+ public String getTableStringIdentifier(ValueInSingleWindow<Row> element) {
+ return element.getValue().getString("dest");
+ }
+ };
+
+ List<Row> rows =
+ ImmutableList.of(
+ Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.table_b").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.table_c").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(4L, "v4", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(5L, "v5", "default.table_b").build());
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(3, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.table_a"));
+ assertTrue(map.containsKey("default.table_b"));
+ assertTrue(map.containsKey("default.table_c"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWindowedDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "t1");
Review Comment:
Did you expect to test same table ID being returned by multiple windows here ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
Review Comment:
Cloud you clarify in the docs how this helper will be used, specially given that this seem to be providing a sample of tables ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3872429868
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
Review Comment:
As-written this effectively acts as a maximum cache size, "sample" is maybe not the right word unless you get more tables than the configured maximum. My lack of experience with Iceberg kind of becomes a problem here, I'm not sure what a "typical" workload looks like in terms of the number of tables being operated on. We could do away with this parameter or make it uncapped by default if that makes more sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5441619553
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/39883?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 58.34%. Comparing base ([`66ecc31`](https://app.codecov.io/gh/apache/beam/commit/66ecc31a22c9fa39dbf70965f17d10b114022185?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`11fffa9`](https://app.codecov.io/gh/apache/beam/commit/11fffa96309ab8d73aba4d4cf7bdec2cf7925f1d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 27 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #39883 +/- ##
============================================
+ Coverage 57.22% 58.34% +1.12%
- Complexity 3661 13472 +9811
============================================
Files 1195 2567 +1372
Lines 193411 268606 +75195
Branches 3831 11030 +7199
============================================
+ Hits 110670 156717 +46047
- Misses 79171 105952 +26781
- Partials 3570 5937 +2367
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/39883/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [java](https://app.codecov.io/gh/apache/beam/pull/39883/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.45% <ø> (-7.75%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/39883?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3918082047
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
We can leave it as a future improvement if we notice it's affecting throughput. I don't think it'll break update compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3908772251
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,834 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
Review Comment:
Can we add a test where a table gets updated across refresh intervals? The consumer should receive the updated metadata after a refresh interval has passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3918584510
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,723 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ private static final TableIdentifier TABLE_ID = TableIdentifier.of("default", "table");
+
+ private static final DynamicDestinations SINGLE_TABLE_DYNAMIC_DESTINATIONS =
+ DynamicDestinations.singleTable(TABLE_ID, BEAM_SCHEMA);
+
+ private static final DynamicDestinations DYNAMIC_DESTINATIONS =
+ new DynamicDestinations() {
+ @Override
+ public Schema getDataSchema() {
+ return BEAM_SCHEMA;
+ }
+
+ @Override
+ public Row getData(Row element) {
+ return element;
+ }
+
+ @Override
+ public IcebergDestination instantiateDestination(String destination) {
+ return IcebergDestination.builder()
+ .setTableIdentifier(IcebergUtils.parseTableIdentifier(destination))
+ .build();
+ }
+
+ @Override
+ public String getTableStringIdentifier(ValueInSingleWindow<Row> element) {
+ return element.getValue().getString("dest");
+ }
+ };
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ Table realTable = getCatalog().createTable(TABLE_ID, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(SINGLE_TABLE_DYNAMIC_DESTINATIONS)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(TABLE_ID);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ List<Row> rows =
+ ImmutableList.of(
+ Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.table_b").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.table_c").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(4L, "v4", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(5L, "v5", "default.table_b").build());
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(3, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.table_a"));
+ assertTrue(map.containsKey("default.table_b"));
+ assertTrue(map.containsKey("default.table_c"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWindowedDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 100; i++) {
+ String dest = (i % 2 == 0) ? "default.t1" : "default.t2";
+ rows.add(Row.withSchema(BEAM_SCHEMA).addValues((long) i, "val_" + i, dest).build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.t1"));
+ assertTrue(map.containsKey("default.t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testUnboundedGlobalWindowStreamingDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "stream_t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "stream_t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.stream_t1").build();
+ Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.stream_t2").build();
+ Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.stream_t1").build();
+
+ TestStream<Row> stream =
+ TestStream.create(RowCoder.of(BEAM_SCHEMA))
+ .advanceWatermarkTo(new Instant(0))
+ .addElements(row1)
+ .addElements(row2)
+ .addElements(row3)
+ .advanceProcessingTime(Duration.standardSeconds(5))
+ .advanceWatermarkToInfinity();
+
+ PCollection<Row> input = pipeline.apply("StreamInput", stream);
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .setRefreshInterval(Duration.standardSeconds(2))
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.stream_t1"));
+ assertTrue(map.containsKey("default.stream_t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMetadataRefreshedAcrossIntervals() {
Review Comment:
This test looks good, but can we please add another similar one consuming the table specifically as a side input? Should also assert the update happened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5500368096
@chamikaramj @ahmedabu98 this should be ready for another review pass!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3909078154
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
Review Comment:
How are we going to control parallelism? Ideally we'd pass tableIds to only a few tables so they can do sequential loadTable calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3872831739
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ TableIdentifier tableId = TableIdentifier.of("default", "single_table");
+ Table realTable = getCatalog().createTable(tableId, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations = DynamicDestinations.singleTable(tableId, BEAM_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(tableId);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ DynamicDestinations dynamicDestinations =
+ new DynamicDestinations() {
+ @Override
+ public Schema getDataSchema() {
+ return BEAM_SCHEMA;
+ }
+
+ @Override
+ public Row getData(Row element) {
+ return element;
+ }
+
+ @Override
+ public IcebergDestination instantiateDestination(String destination) {
+ return IcebergDestination.builder()
+ .setTableIdentifier(IcebergUtils.parseTableIdentifier(destination))
+ .build();
+ }
+
+ @Override
+ public String getTableStringIdentifier(ValueInSingleWindow<Row> element) {
+ return element.getValue().getString("dest");
+ }
+ };
+
+ List<Row> rows =
+ ImmutableList.of(
+ Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.table_b").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.table_c").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(4L, "v4", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(5L, "v5", "default.table_b").build());
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(3, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.table_a"));
+ assertTrue(map.containsKey("default.table_b"));
+ assertTrue(map.containsKey("default.table_c"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWindowedDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "t1");
Review Comment:
This is what I get for leaning on Gemini to produce some unit tests. The name is misleading, it's really testing that we deduplicate the target table IDs when we build the spec (since there are 50 elements referring to t1 and 50 elements referring to t2, we deduplicate that to single references to the two tables.) I'll rename this, the window doesn't really matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3937620859
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -337,7 +382,57 @@ public void processElement(
"Table '{}' does not exist in catalog. Skipping metadata emission for side-input view.",
tableIdString);
TABLES_SKIPPED_MISSING_COUNTER.inc();
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Skipping metadata emission for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
}
}
}
+
+ static class AccumulateTableMetadataMapDoFn
+ extends DoFn<
+ KV<Void, KV<String, SerializableTableSpec>>, Map<String, SerializableTableSpec>> {
+ @StateId("tableCache")
+ private final StateSpec<MapState<String, SerializableTableSpec>> cacheStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), SerializableTableSpec.getCoder());
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<Void, KV<String, SerializableTableSpec>> element,
+ @StateId("tableCache") MapState<String, SerializableTableSpec> cacheState,
+ OutputReceiver<Map<String, SerializableTableSpec>> out) {
+ KV<String, SerializableTableSpec> kv = element.getValue();
+ cacheState.put(kv.getKey(), kv.getValue());
+
+ Map<String, SerializableTableSpec> mapSnapshot = new HashMap<>();
+ for (Map.Entry<String, SerializableTableSpec> entry : cacheState.entries().read()) {
+ mapSnapshot.put(entry.getKey(), entry.getValue());
+ }
+ out.output(Collections.unmodifiableMap(mapSnapshot));
+ }
+ }
+
+ static class MapMergerFn extends Combine.BinaryCombineFn<Map<String, SerializableTableSpec>> {
+ @Override
+ public Map<String, SerializableTableSpec> apply(
+ Map<String, SerializableTableSpec> left, Map<String, SerializableTableSpec> right) {
+ if (left == null || left.isEmpty()) {
+ return right != null ? right : Collections.emptyMap();
+ }
+ if (right == null || right.isEmpty()) {
+ return left;
+ }
+ Map<String, SerializableTableSpec> merged = new HashMap<>(left);
+ merged.putAll(right);
Review Comment:
Are we sure that `right` will always be the newer metadata?
I think we should store `TableMetadata.lastUpdatedMillis()` in `SerializableTableSpec` and use it to compare `right` and `left` to choose the freshest metadata
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -337,7 +382,57 @@ public void processElement(
"Table '{}' does not exist in catalog. Skipping metadata emission for side-input view.",
tableIdString);
TABLES_SKIPPED_MISSING_COUNTER.inc();
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Skipping metadata emission for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
}
}
}
+
+ static class AccumulateTableMetadataMapDoFn
+ extends DoFn<
+ KV<Void, KV<String, SerializableTableSpec>>, Map<String, SerializableTableSpec>> {
+ @StateId("tableCache")
+ private final StateSpec<MapState<String, SerializableTableSpec>> cacheStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), SerializableTableSpec.getCoder());
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<Void, KV<String, SerializableTableSpec>> element,
+ @StateId("tableCache") MapState<String, SerializableTableSpec> cacheState,
+ OutputReceiver<Map<String, SerializableTableSpec>> out) {
+ KV<String, SerializableTableSpec> kv = element.getValue();
+ cacheState.put(kv.getKey(), kv.getValue());
+
+ Map<String, SerializableTableSpec> mapSnapshot = new HashMap<>();
+ for (Map.Entry<String, SerializableTableSpec> entry : cacheState.entries().read()) {
+ mapSnapshot.put(entry.getKey(), entry.getValue());
Review Comment:
I think state cache is a good idea, but we should try clean up old tables otherwise state and side-input will get unnecessarily bloated.
Might help to store a separate "last seen" millis next to each table, and renew it every time a new table comes thru here. We can use it to identify idle tables (older than `refreshInterval`) and remove them from the cache
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -337,7 +382,57 @@ public void processElement(
"Table '{}' does not exist in catalog. Skipping metadata emission for side-input view.",
tableIdString);
TABLES_SKIPPED_MISSING_COUNTER.inc();
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Skipping metadata emission for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
Review Comment:
Can we break up this try block into separate blocks? `SerializableTableSpec.fromTable` can throw `IllegalArgumentException`, which this code would log as a table identifier parsing error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3887379992
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, samples up to a maximum number of tables, loads their declarative metadata from the
+ * Iceberg catalog, and emits {@link KV} pairs of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. If the number of distinct tables in a window
+ * exceeds {@code maxTables}, up to {@code maxTables} tables are sampled into the broadcasted view,
+ * while remaining destinations can fall back to worker-local catalog loading.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final int DEFAULT_MAX_TABLES = 100;
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract int getMaxTables();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder().setMaxTables(DEFAULT_MAX_TABLES);
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaxTables(int maxTables);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Preconditions.checkArgument(
+ driver.getMaxTables() > 0,
+ "maxTables must be greater than 0, got %s",
+ driver.getMaxTables());
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates a {@link PCollectionView} of {@link
+ * Map} of table identifier strings to {@link SerializableTableSpec} using {@link
+ * #DEFAULT_MAX_TABLES}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, DEFAULT_MAX_TABLES);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with a custom {@code maxTables} limit and
+ * creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maxTables maximum distinct tables to poll and broadcast per window.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ int maxTables) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaxTables(maxTables)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ PCollection<String> distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
nonblocking: is this meant to support default streaming inputs? `Distinct` rejects an unbounded `GlobalWindow` with default trigger. Iceberg write accepts that input shape before its own triggering. fine if this is batch/pre-windowed only, but worth making boundary explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #39883:
URL: https://github.com/apache/beam/pull/39883#issuecomment-5494971623
CC: @ahmedabu98
Currently exploring if moving to another method of deduplication makes sense, the original design doc specifically called out using Distinct but also wants unbounded global window streaming support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3904811776
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, samples up to a maximum number of tables, loads their declarative metadata from the
+ * Iceberg catalog, and emits {@link KV} pairs of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. If the number of distinct tables in a window
+ * exceeds {@code maxTables}, up to {@code maxTables} tables are sampled into the broadcasted view,
+ * while remaining destinations can fall back to worker-local catalog loading.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final int DEFAULT_MAX_TABLES = 100;
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract int getMaxTables();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder().setMaxTables(DEFAULT_MAX_TABLES);
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaxTables(int maxTables);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Preconditions.checkArgument(
+ driver.getMaxTables() > 0,
+ "maxTables must be greater than 0, got %s",
+ driver.getMaxTables());
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates a {@link PCollectionView} of {@link
+ * Map} of table identifier strings to {@link SerializableTableSpec} using {@link
+ * #DEFAULT_MAX_TABLES}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, DEFAULT_MAX_TABLES);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with a custom {@code maxTables} limit and
+ * creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maxTables maximum distinct tables to poll and broadcast per window.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ int maxTables) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaxTables(maxTables)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ PCollection<String> distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> sampledTableIds =
+ distinctTableIds.apply("SampleTableIds", Sample.any(getMaxTables()));
+
+ return sampledTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn extends DoFn<String, KV<String, SerializableTableSpec>> {
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString, OutputReceiver<KV<String, SerializableTableSpec>> out) {
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ Table table = catalogConfig.catalog().loadTable(tableId);
Review Comment:
Going back to the original design doc, Ahmed had suggested catching NoSuchTableExceptions gracefully since the lack of metadata tells the sink to try creating the table. I've added that logic, which should avoid failing whole bundles when we get an expected exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3908688279
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.addIfNotNull(
+ DisplayData.item("maximumCacheSize", getMaximumCacheSize())
+ .withLabel("Maximum Cache Size"));
+ builder.addIfNotNull(
+ DisplayData.item("refreshInterval", getRefreshInterval())
+ .withLabel("Table Metadata Refresh Interval"));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn extends DoFn<String, KV<String, SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(CatalogPollingDoFn.class);
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+ private static final Counter TABLES_SKIPPED_MISSING_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSkippedMissing");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString, OutputReceiver<KV<String, SerializableTableSpec>> out) {
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ try {
+ Table table = catalogConfig.catalog().loadTable(tableId);
Review Comment:
Let's use `TableCache.getAndRefreshIfStale()`
The method uses a default refresh interval. Feel free to adjust the API surface to pass in the custom refresh interval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3872724811
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, samples up to a maximum number of tables, loads their declarative metadata from the
+ * Iceberg catalog, and emits {@link KV} pairs of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. If the number of distinct tables in a window
+ * exceeds {@code maxTables}, up to {@code maxTables} tables are sampled into the broadcasted view,
+ * while remaining destinations can fall back to worker-local catalog loading.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final int DEFAULT_MAX_TABLES = 100;
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract int getMaxTables();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder().setMaxTables(DEFAULT_MAX_TABLES);
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaxTables(int maxTables);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Preconditions.checkArgument(
+ driver.getMaxTables() > 0,
+ "maxTables must be greater than 0, got %s",
+ driver.getMaxTables());
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates a {@link PCollectionView} of {@link
+ * Map} of table identifier strings to {@link SerializableTableSpec} using {@link
+ * #DEFAULT_MAX_TABLES}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, DEFAULT_MAX_TABLES);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with a custom {@code maxTables} limit and
+ * creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maxTables maximum distinct tables to poll and broadcast per window.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ int maxTables) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaxTables(maxTables)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ PCollection<String> distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> sampledTableIds =
+ distinctTableIds.apply("SampleTableIds", Sample.any(getMaxTables()));
+
+ return sampledTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn extends DoFn<String, KV<String, SerializableTableSpec>> {
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString, OutputReceiver<KV<String, SerializableTableSpec>> out) {
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ Table table = catalogConfig.catalog().loadTable(tableId);
Review Comment:
We could set up some sort of DLQ here. As best I can tell the current iceberg implementation uses the same approach of outright failing on bad table identifiers though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3875238830
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.joda.time.Instant;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
Review Comment:
I was just using the wording from the existing comment :)
I think if we are trying to cache maximum possible for efficiency, but rest will still work (less efficiently), it makes sense. In general, lets's expand more about downstream use-case here to clarify what it's intended for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3926012784
##########
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriverTest.java:
##########
@@ -0,0 +1,723 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionKey;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TableMetadataDriverTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private String warehouseLocation;
+ private IcebergCatalogConfig catalogConfig;
+
+ private static final Schema BEAM_SCHEMA =
+ Schema.builder()
+ .addInt64Field("id")
+ .addStringField("data")
+ .addNullableStringField("dest")
+ .build();
+
+ private static final org.apache.iceberg.Schema ICEBERG_SCHEMA =
+ IcebergUtils.beamSchemaToIcebergSchema(
+ Schema.builder().addInt64Field("id").addStringField("data").build());
+
+ private static final TableIdentifier TABLE_ID = TableIdentifier.of("default", "table");
+
+ private static final DynamicDestinations SINGLE_TABLE_DYNAMIC_DESTINATIONS =
+ DynamicDestinations.singleTable(TABLE_ID, BEAM_SCHEMA);
+
+ private static final DynamicDestinations DYNAMIC_DESTINATIONS =
+ new DynamicDestinations() {
+ @Override
+ public Schema getDataSchema() {
+ return BEAM_SCHEMA;
+ }
+
+ @Override
+ public Row getData(Row element) {
+ return element;
+ }
+
+ @Override
+ public IcebergDestination instantiateDestination(String destination) {
+ return IcebergDestination.builder()
+ .setTableIdentifier(IcebergUtils.parseTableIdentifier(destination))
+ .build();
+ }
+
+ @Override
+ public String getTableStringIdentifier(ValueInSingleWindow<Row> element) {
+ return element.getValue().getString("dest");
+ }
+ };
+
+ @Before
+ public void setUp() throws Exception {
+ warehouseLocation = "file:" + tempFolder.newFolder().getAbsolutePath();
+ catalogConfig =
+ IcebergCatalogConfig.builder()
+ .setCatalogName("hadoop")
+ .setCatalogProperties(ImmutableMap.of("type", "hadoop", "warehouse", warehouseLocation))
+ .build();
+ }
+
+ private Catalog getCatalog() {
+ return CatalogUtil.loadCatalog(
+ CatalogUtil.ICEBERG_CATALOG_HADOOP,
+ "hadoop",
+ ImmutableMap.of(CatalogProperties.WAREHOUSE_LOCATION, warehouseLocation),
+ new Configuration());
+ }
+
+ @Test
+ public void testSingleTableExtractionAndSpecOutput() {
+ Table realTable = getCatalog().createTable(TABLE_ID, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ rows.add(
+ Row.withSchema(BEAM_SCHEMA)
+ .withFieldValue("id", (long) i)
+ .withFieldValue("data", "val_" + i)
+ .withFieldValue("dest", null)
+ .build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(SINGLE_TABLE_DYNAMIC_DESTINATIONS)
+ .build());
+
+ String expectedTableIdString = IcebergUtils.tableIdentifierToString(TABLE_ID);
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(1, list.size());
+ KV<String, SerializableTableSpec> kv = list.get(0);
+ assertEquals(expectedTableIdString, kv.getKey());
+ SerializableTableSpec spec = kv.getValue();
+ assertNotNull(spec);
+ assertEquals(realTable.name(), spec.getName());
+ assertEquals(realTable.location(), spec.getLocation());
+ assertEquals(realTable.schema().asStruct(), spec.getSchema().asStruct());
+ assertEquals(realTable.spec(), spec.getPartitionSpec());
+ assertNotNull(spec.getFileIO());
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMultipleDynamicDestinationsExtraction() {
+ Catalog catalog = getCatalog();
+ TableIdentifier tableA = TableIdentifier.of("default", "table_a");
+ TableIdentifier tableB = TableIdentifier.of("default", "table_b");
+ TableIdentifier tableC = TableIdentifier.of("default", "table_c");
+
+ catalog.createTable(tableA, ICEBERG_SCHEMA);
+ catalog.createTable(tableB, ICEBERG_SCHEMA);
+ catalog.createTable(tableC, ICEBERG_SCHEMA);
+
+ List<Row> rows =
+ ImmutableList.of(
+ Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.table_b").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.table_c").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(4L, "v4", "default.table_a").build(),
+ Row.withSchema(BEAM_SCHEMA).addValues(5L, "v5", "default.table_b").build());
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(3, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.table_a"));
+ assertTrue(map.containsKey("default.table_b"));
+ assertTrue(map.containsKey("default.table_c"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWindowedDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ List<Row> rows = new ArrayList<>();
+ for (int i = 0; i < 100; i++) {
+ String dest = (i % 2 == 0) ? "default.t1" : "default.t2";
+ rows.add(Row.withSchema(BEAM_SCHEMA).addValues((long) i, "val_" + i, dest).build());
+ }
+
+ PCollection<Row> input = pipeline.apply(Create.of(rows)).setCoder(RowCoder.of(BEAM_SCHEMA));
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.t1"));
+ assertTrue(map.containsKey("default.t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testUnboundedGlobalWindowStreamingDeduplication() {
+ Catalog catalog = getCatalog();
+ TableIdentifier table1 = TableIdentifier.of("default", "stream_t1");
+ TableIdentifier table2 = TableIdentifier.of("default", "stream_t2");
+
+ catalog.createTable(table1, ICEBERG_SCHEMA);
+ catalog.createTable(table2, ICEBERG_SCHEMA);
+
+ Row row1 = Row.withSchema(BEAM_SCHEMA).addValues(1L, "v1", "default.stream_t1").build();
+ Row row2 = Row.withSchema(BEAM_SCHEMA).addValues(2L, "v2", "default.stream_t2").build();
+ Row row3 = Row.withSchema(BEAM_SCHEMA).addValues(3L, "v3", "default.stream_t1").build();
+
+ TestStream<Row> stream =
+ TestStream.create(RowCoder.of(BEAM_SCHEMA))
+ .advanceWatermarkTo(new Instant(0))
+ .addElements(row1)
+ .addElements(row2)
+ .addElements(row3)
+ .advanceProcessingTime(Duration.standardSeconds(5))
+ .advanceWatermarkToInfinity();
+
+ PCollection<Row> input = pipeline.apply("StreamInput", stream);
+
+ PCollection<KV<String, SerializableTableSpec>> specs =
+ input.apply(
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(DYNAMIC_DESTINATIONS)
+ .setRefreshInterval(Duration.standardSeconds(2))
+ .build());
+
+ PAssert.that(specs)
+ .satisfies(
+ elements -> {
+ List<KV<String, SerializableTableSpec>> list = ImmutableList.copyOf(elements);
+ assertEquals(2, list.size());
+ Map<String, SerializableTableSpec> map =
+ list.stream().collect(ImmutableMap.toImmutableMap(KV::getKey, KV::getValue));
+ assertTrue(map.containsKey("default.stream_t1"));
+ assertTrue(map.containsKey("default.stream_t2"));
+ return null;
+ });
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testMetadataRefreshedAcrossIntervals() {
Review Comment:
This was a good call, adding that test variation found a number of problems with streaming cases and the triggering setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3908688279
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ return cachedTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(KvCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()));
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.addIfNotNull(
+ DisplayData.item("maximumCacheSize", getMaximumCacheSize())
+ .withLabel("Maximum Cache Size"));
+ builder.addIfNotNull(
+ DisplayData.item("refreshInterval", getRefreshInterval())
+ .withLabel("Table Metadata Refresh Interval"));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn extends DoFn<String, KV<String, SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(CatalogPollingDoFn.class);
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+ private static final Counter TABLES_SKIPPED_MISSING_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSkippedMissing");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString, OutputReceiver<KV<String, SerializableTableSpec>> out) {
+ TableIdentifier tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ try {
+ Table table = catalogConfig.catalog().loadTable(tableId);
Review Comment:
Let's use `TableCache.getAndRefreshIfStale()`
The method uses a default refresh interval. Feel free to adjust the API surface to pass in the custom refresh interval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3916974149
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize}, loads their
+ * declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of table identifier
+ * strings to {@link SerializableTableSpec}. This is intended to be used in Beam pipelines that may
+ * utilize a large number of workers to handle Iceberg writes, where having every worker thread
+ * query for table metadata results in an excessive amount of requests and a high level of
+ * redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, an {@link AfterProcessingTime}
+ * trigger is automatically applied to fire deduplication and refresh table metadata at the
+ * configured {@code refreshInterval} (defaulting to {@link #DEFAULT_REFRESH_INTERVAL}).
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(refreshInterval)
+ .build())
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of());
+
+ boolean isUnboundedGlobal =
+ input.isBounded() == PCollection.IsBounded.UNBOUNDED
+ && input.getWindowingStrategy().getWindowFn() instanceof GlobalWindows;
+
+ PCollection<String> triggeredTableIds;
+ if (isUnboundedGlobal) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ triggeredTableIds =
+ tableIds.apply(
+ "ApplyStreamingTrigger",
+ Window.<String>into(new GlobalWindows())
+ .triggering(
+ Repeatedly.forever(
+ AfterProcessingTime.pastFirstElementInPane().plusDelayOf(interval)))
+ .accumulatingFiredPanes());
+ } else {
+ triggeredTableIds = tableIds;
+ }
+
+ PCollection<String> distinctTableIds =
+ triggeredTableIds.apply("DistinctTableIds", Distinct.create());
Review Comment:
Yeah Distinct only emits once per window, so more batch-like.
AFAICT Deduplicate emits once and stores the value in state. It won't emit the value again while it's in state. After `refreshInterval` time passes, it clears the value and allows a new encounter to be emitted once again.
> Deduplicate is a stateful DoFn, which the initial problem statement and design scope seemed insistent on avoiding
That's a good point cuz Deduplicate will still have one thread per tableId.
I think we can relieve some of the pressure here by having the upstream (pre-bottleneck) `ExtractTableIdsDoFn` transform do some local filtering. It can keep track of when tableIds where last emitted (e.g. Map<String, Instant>) and only output a tableId if it's older than `refreshInterval / 2`. Doesn't have to be exact or durable, if duplicates still get through then stateful Deduplicate will take care of them.
Also I think this bottleneck is a lot more gentle than the one the doc mentions. We're passing table strings through a side input instead of full data rows through the main path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #39883:
URL: https://github.com/apache/beam/pull/39883#discussion_r3938855157
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.state.MapState;
+import org.apache.beam.sdk.state.ReadableState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Deduplicate;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.StateId;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize} (batch pipelines
+ * only), loads their declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of
+ * table identifier strings to {@link SerializableTableSpec} (or {@code null} if the table does not
+ * exist or fails to load). This is intended to be used in Beam pipelines that may utilize a large
+ * number of workers to handle Iceberg writes, where having every worker thread query for table
+ * metadata results in an excessive amount of requests and a high level of redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading. Note that {@code maximumCacheSize} is
+ * currently supported for bounded batch pipelines only.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, {@link Deduplicate} is used to
+ * deduplicate table identifiers over the configured {@code refreshInterval} (defaulting to {@link
+ * #DEFAULT_REFRESH_INTERVAL}), allowing periodic refresh of table metadata when schemas evolve.
+ * Missing table signals ({@code null} specs) trigger side-input view materialization without
+ * caching the missing tables, ensuring downstream consumers are never blocked waiting for the side
+ * input.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, @Nullable SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+ public static final int DEFAULT_POLLING_BUCKETS = 1;
+
+ @FunctionalInterface
+ public interface Clock extends Serializable {
+ long currentTimeMillis();
+ }
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ /**
+ * Returns the number of parallel buckets/workers used to query the Iceberg catalog, or {@code
+ * null} for default.
+ */
+ public abstract @Nullable Integer getPollingBuckets();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ /**
+ * Sets the number of parallel buckets (worker tasks) used to query the Iceberg catalog.
+ *
+ * <p>Defaults to {@link #DEFAULT_POLLING_BUCKETS} (1), which serializes all catalog lookups to
+ * avoid overwhelming catalog metastores (e.g. Hive Metastore, REST catalog). For pipelines
+ * writing to a large number of distinct dynamic tables (e.g. hundreds of tables per window),
+ * consider increasing this value (e.g. 5–10) to parallelize catalog lookups while still
+ * bounding load.
+ */
+ public abstract Builder setPollingBuckets(@Nullable Integer pollingBuckets);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ Integer pollingBuckets = driver.getPollingBuckets();
+ if (pollingBuckets != null) {
+ Preconditions.checkArgument(
+ pollingBuckets > 0, "pollingBuckets must be greater than 0, got %s", pollingBuckets);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null, null);
+ }
Review Comment:
nit: these overloads are getting pretty big 😅
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.state.MapState;
+import org.apache.beam.sdk.state.ReadableState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Deduplicate;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.StateId;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize} (batch pipelines
+ * only), loads their declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of
+ * table identifier strings to {@link SerializableTableSpec} (or {@code null} if the table does not
+ * exist or fails to load). This is intended to be used in Beam pipelines that may utilize a large
+ * number of workers to handle Iceberg writes, where having every worker thread query for table
+ * metadata results in an excessive amount of requests and a high level of redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading. Note that {@code maximumCacheSize} is
+ * currently supported for bounded batch pipelines only.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, {@link Deduplicate} is used to
+ * deduplicate table identifiers over the configured {@code refreshInterval} (defaulting to {@link
+ * #DEFAULT_REFRESH_INTERVAL}), allowing periodic refresh of table metadata when schemas evolve.
+ * Missing table signals ({@code null} specs) trigger side-input view materialization without
+ * caching the missing tables, ensuring downstream consumers are never blocked waiting for the side
+ * input.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, @Nullable SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+ public static final int DEFAULT_POLLING_BUCKETS = 1;
+
+ @FunctionalInterface
+ public interface Clock extends Serializable {
+ long currentTimeMillis();
+ }
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ /**
+ * Returns the number of parallel buckets/workers used to query the Iceberg catalog, or {@code
+ * null} for default.
+ */
+ public abstract @Nullable Integer getPollingBuckets();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ /**
+ * Sets the number of parallel buckets (worker tasks) used to query the Iceberg catalog.
+ *
+ * <p>Defaults to {@link #DEFAULT_POLLING_BUCKETS} (1), which serializes all catalog lookups to
+ * avoid overwhelming catalog metastores (e.g. Hive Metastore, REST catalog). For pipelines
+ * writing to a large number of distinct dynamic tables (e.g. hundreds of tables per window),
+ * consider increasing this value (e.g. 5–10) to parallelize catalog lookups while still
+ * bounding load.
+ */
+ public abstract Builder setPollingBuckets(@Nullable Integer pollingBuckets);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ Integer pollingBuckets = driver.getPollingBuckets();
+ if (pollingBuckets != null) {
+ Preconditions.checkArgument(
+ pollingBuckets > 0, "pollingBuckets must be greater than 0, got %s", pollingBuckets);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, refreshInterval, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize}
+ * limit, custom {@code refreshInterval}, and custom {@code pollingBuckets}, creating a {@link
+ * PCollectionView} of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ * @param pollingBuckets optional number of parallel buckets/workers for catalog polling.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Clock clock) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null,
+ clock);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Duration cacheTtl,
+ @Nullable Clock clock) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ Duration interval = refreshInterval != null ? refreshInterval : DEFAULT_REFRESH_INTERVAL;
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ input.apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(interval)
+ .setPollingBuckets(pollingBuckets)
+ .build());
+
+ if (isStreaming) {
+ AccumulateTableMetadataMapDoFn accumulateDoFn =
+ clock != null
+ ? new AccumulateTableMetadataMapDoFn(interval, cacheTtl, clock)
+ : new AccumulateTableMetadataMapDoFn(interval, cacheTtl);
+ return specs
+ .apply("KeyForGlobalCache", WithKeys.of((Void) null))
+ .setCoder(KvCoder.of(VoidCoder.of(), specs.getCoder()))
+ .apply("AccumulateCacheMap", ParDo.of(accumulateDoFn))
+ .setCoder(MapCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()))
+ .apply(
+ "StreamingCacheWindow",
+ Window.<Map<String, SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes())
+ .apply(
+ "CreateMetadataSingletonView",
+ Combine.globally(new MapMergerFn()).asSingletonView());
+ }
+
+ return specs
+ .apply(
+ "FilterValidSpecsForView",
+ Filter.by(
+ (SerializableFunction<KV<String, @Nullable SerializableTableSpec>, Boolean>)
+ kv -> kv.getValue() != null))
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, @Nullable SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of())
+ .apply("MetadataGlobalWindow", Window.into(new GlobalWindows()));
+
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ PCollection<String> distinctTableIds;
+ if (isStreaming) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ distinctTableIds =
+ tableIds.apply(
+ "DeduplicateTableIds", Deduplicate.<String>values().withDuration(interval));
+ } else {
+ distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+ }
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ if (isStreaming) {
+ throw new UnsupportedOperationException(
+ "maximumCacheSize is currently not supported for unbounded streaming pipelines.");
+ }
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ @Nullable Integer configuredBuckets = getPollingBuckets();
+ int pollingBuckets = configuredBuckets != null ? configuredBuckets : DEFAULT_POLLING_BUCKETS;
+ PCollection<String> pollingTableIds =
+ cachedTableIds.apply(
+ "ReshufflePollingBuckets",
+ Reshuffle.<String>viaRandomKey().withNumBuckets(pollingBuckets));
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ pollingTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(
+ KvCoder.of(
+ StringUtf8Coder.of(), NullableCoder.of(SerializableTableSpec.getCoder())));
+
+ if (isStreaming) {
+ return specs.apply(
+ "ApplyStreamingTrigger",
+ Window.<KV<String, @Nullable SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes());
+ }
+ return specs;
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.addIfNotNull(
+ DisplayData.item("maximumCacheSize", getMaximumCacheSize())
+ .withLabel("Maximum Cache Size"));
+ builder.addIfNotNull(
+ DisplayData.item("refreshInterval", getRefreshInterval())
+ .withLabel("Table Metadata Refresh Interval"));
+ builder.addIfNotNull(
+ DisplayData.item("pollingBuckets", getPollingBuckets())
+ .withLabel("Catalog Polling Buckets"));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn
+ extends DoFn<String, KV<String, @Nullable SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(CatalogPollingDoFn.class);
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+ private static final Counter TABLES_SKIPPED_MISSING_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSkippedMissing");
+ private static final Counter TABLES_PARSE_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesParseFailed");
+ private static final Counter TABLES_SPEC_CREATION_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSpecCreationFailed");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString,
+ OutputReceiver<KV<String, @Nullable SerializableTableSpec>> out) {
+ TableIdentifier tableId;
+ try {
+ tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_PARSE_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+
+ Table table;
+ try {
+ table = catalogConfig.catalog().loadTable(tableId);
+ } catch (NoSuchTableException e) {
+ LOG.info(
+ "Table '{}' does not exist in catalog. Emitting empty metadata signal for side-input view.",
+ tableIdString);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ SerializableTableSpec spec;
+ try {
+ spec = SerializableTableSpec.fromTable(tableIdString, table);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to create SerializableTableSpec for table '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SPEC_CREATION_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ TABLES_POLLED_COUNTER.inc();
+ out.output(KV.of(tableIdString, spec));
+ }
+ }
+
+ static class AccumulateTableMetadataMapDoFn
+ extends DoFn<
+ KV<Void, KV<String, @Nullable SerializableTableSpec>>,
+ Map<String, SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(AccumulateTableMetadataMapDoFn.class);
+ private static final Counter TABLES_EVICTED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesEvictedUnused");
+ static final int DEFAULT_TTL_MULTIPLIER = 3;
+
+ @StateId("tableCache")
+ private final StateSpec<MapState<String, SerializableTableSpec>> cacheStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), SerializableTableSpec.getCoder());
+
+ @StateId("lastSeen")
+ private final StateSpec<MapState<String, Long>> lastSeenStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), VarLongCoder.of());
+
+ private final Duration refreshInterval;
+ private final Duration cacheTtl;
+ private final Clock clock;
+
+ AccumulateTableMetadataMapDoFn() {
+ this(DEFAULT_REFRESH_INTERVAL, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval) {
+ this(refreshInterval, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval, Clock clock) {
+ this(refreshInterval, null, clock);
+ }
Review Comment:
nit: cleanup unused constructors
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.state.MapState;
+import org.apache.beam.sdk.state.ReadableState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Deduplicate;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.StateId;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize} (batch pipelines
+ * only), loads their declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of
+ * table identifier strings to {@link SerializableTableSpec} (or {@code null} if the table does not
+ * exist or fails to load). This is intended to be used in Beam pipelines that may utilize a large
+ * number of workers to handle Iceberg writes, where having every worker thread query for table
+ * metadata results in an excessive amount of requests and a high level of redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading. Note that {@code maximumCacheSize} is
+ * currently supported for bounded batch pipelines only.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, {@link Deduplicate} is used to
+ * deduplicate table identifiers over the configured {@code refreshInterval} (defaulting to {@link
+ * #DEFAULT_REFRESH_INTERVAL}), allowing periodic refresh of table metadata when schemas evolve.
+ * Missing table signals ({@code null} specs) trigger side-input view materialization without
+ * caching the missing tables, ensuring downstream consumers are never blocked waiting for the side
+ * input.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, @Nullable SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+ public static final int DEFAULT_POLLING_BUCKETS = 1;
+
+ @FunctionalInterface
+ public interface Clock extends Serializable {
+ long currentTimeMillis();
+ }
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ /**
+ * Returns the number of parallel buckets/workers used to query the Iceberg catalog, or {@code
+ * null} for default.
+ */
+ public abstract @Nullable Integer getPollingBuckets();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ /**
+ * Sets the number of parallel buckets (worker tasks) used to query the Iceberg catalog.
+ *
+ * <p>Defaults to {@link #DEFAULT_POLLING_BUCKETS} (1), which serializes all catalog lookups to
+ * avoid overwhelming catalog metastores (e.g. Hive Metastore, REST catalog). For pipelines
+ * writing to a large number of distinct dynamic tables (e.g. hundreds of tables per window),
+ * consider increasing this value (e.g. 5–10) to parallelize catalog lookups while still
+ * bounding load.
+ */
+ public abstract Builder setPollingBuckets(@Nullable Integer pollingBuckets);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ Integer pollingBuckets = driver.getPollingBuckets();
+ if (pollingBuckets != null) {
+ Preconditions.checkArgument(
+ pollingBuckets > 0, "pollingBuckets must be greater than 0, got %s", pollingBuckets);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, refreshInterval, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize}
+ * limit, custom {@code refreshInterval}, and custom {@code pollingBuckets}, creating a {@link
+ * PCollectionView} of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ * @param pollingBuckets optional number of parallel buckets/workers for catalog polling.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Clock clock) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null,
+ clock);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Duration cacheTtl,
+ @Nullable Clock clock) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ Duration interval = refreshInterval != null ? refreshInterval : DEFAULT_REFRESH_INTERVAL;
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ input.apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(interval)
+ .setPollingBuckets(pollingBuckets)
+ .build());
+
+ if (isStreaming) {
+ AccumulateTableMetadataMapDoFn accumulateDoFn =
+ clock != null
+ ? new AccumulateTableMetadataMapDoFn(interval, cacheTtl, clock)
+ : new AccumulateTableMetadataMapDoFn(interval, cacheTtl);
+ return specs
+ .apply("KeyForGlobalCache", WithKeys.of((Void) null))
+ .setCoder(KvCoder.of(VoidCoder.of(), specs.getCoder()))
+ .apply("AccumulateCacheMap", ParDo.of(accumulateDoFn))
+ .setCoder(MapCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()))
+ .apply(
+ "StreamingCacheWindow",
+ Window.<Map<String, SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes())
+ .apply(
+ "CreateMetadataSingletonView",
+ Combine.globally(new MapMergerFn()).asSingletonView());
+ }
+
+ return specs
+ .apply(
+ "FilterValidSpecsForView",
+ Filter.by(
+ (SerializableFunction<KV<String, @Nullable SerializableTableSpec>, Boolean>)
+ kv -> kv.getValue() != null))
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, @Nullable SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of())
+ .apply("MetadataGlobalWindow", Window.into(new GlobalWindows()));
+
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ PCollection<String> distinctTableIds;
+ if (isStreaming) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ distinctTableIds =
+ tableIds.apply(
+ "DeduplicateTableIds", Deduplicate.<String>values().withDuration(interval));
+ } else {
+ distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+ }
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ if (isStreaming) {
+ throw new UnsupportedOperationException(
+ "maximumCacheSize is currently not supported for unbounded streaming pipelines.");
+ }
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ @Nullable Integer configuredBuckets = getPollingBuckets();
+ int pollingBuckets = configuredBuckets != null ? configuredBuckets : DEFAULT_POLLING_BUCKETS;
+ PCollection<String> pollingTableIds =
+ cachedTableIds.apply(
+ "ReshufflePollingBuckets",
+ Reshuffle.<String>viaRandomKey().withNumBuckets(pollingBuckets));
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ pollingTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(
+ KvCoder.of(
+ StringUtf8Coder.of(), NullableCoder.of(SerializableTableSpec.getCoder())));
+
+ if (isStreaming) {
+ return specs.apply(
+ "ApplyStreamingTrigger",
+ Window.<KV<String, @Nullable SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes());
+ }
+ return specs;
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.addIfNotNull(
+ DisplayData.item("maximumCacheSize", getMaximumCacheSize())
+ .withLabel("Maximum Cache Size"));
+ builder.addIfNotNull(
+ DisplayData.item("refreshInterval", getRefreshInterval())
+ .withLabel("Table Metadata Refresh Interval"));
+ builder.addIfNotNull(
+ DisplayData.item("pollingBuckets", getPollingBuckets())
+ .withLabel("Catalog Polling Buckets"));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn
+ extends DoFn<String, KV<String, @Nullable SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(CatalogPollingDoFn.class);
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+ private static final Counter TABLES_SKIPPED_MISSING_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSkippedMissing");
+ private static final Counter TABLES_PARSE_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesParseFailed");
+ private static final Counter TABLES_SPEC_CREATION_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSpecCreationFailed");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString,
+ OutputReceiver<KV<String, @Nullable SerializableTableSpec>> out) {
+ TableIdentifier tableId;
+ try {
+ tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_PARSE_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+
+ Table table;
+ try {
+ table = catalogConfig.catalog().loadTable(tableId);
+ } catch (NoSuchTableException e) {
+ LOG.info(
+ "Table '{}' does not exist in catalog. Emitting empty metadata signal for side-input view.",
+ tableIdString);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ SerializableTableSpec spec;
+ try {
+ spec = SerializableTableSpec.fromTable(tableIdString, table);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to create SerializableTableSpec for table '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SPEC_CREATION_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ TABLES_POLLED_COUNTER.inc();
+ out.output(KV.of(tableIdString, spec));
+ }
+ }
+
+ static class AccumulateTableMetadataMapDoFn
+ extends DoFn<
+ KV<Void, KV<String, @Nullable SerializableTableSpec>>,
+ Map<String, SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(AccumulateTableMetadataMapDoFn.class);
+ private static final Counter TABLES_EVICTED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesEvictedUnused");
+ static final int DEFAULT_TTL_MULTIPLIER = 3;
+
+ @StateId("tableCache")
+ private final StateSpec<MapState<String, SerializableTableSpec>> cacheStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), SerializableTableSpec.getCoder());
+
+ @StateId("lastSeen")
+ private final StateSpec<MapState<String, Long>> lastSeenStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), VarLongCoder.of());
+
+ private final Duration refreshInterval;
+ private final Duration cacheTtl;
+ private final Clock clock;
+
+ AccumulateTableMetadataMapDoFn() {
+ this(DEFAULT_REFRESH_INTERVAL, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval) {
+ this(refreshInterval, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval, Clock clock) {
+ this(refreshInterval, null, clock);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval, @Nullable Duration cacheTtl) {
+ this(refreshInterval, cacheTtl, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(
+ Duration refreshInterval, @Nullable Duration cacheTtl, Clock clock) {
+ this.refreshInterval = refreshInterval != null ? refreshInterval : DEFAULT_REFRESH_INTERVAL;
+ this.cacheTtl =
+ cacheTtl != null ? cacheTtl : this.refreshInterval.multipliedBy(DEFAULT_TTL_MULTIPLIER);
+ this.clock = clock != null ? clock : System::currentTimeMillis;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<Void, KV<String, @Nullable SerializableTableSpec>> element,
+ @StateId("tableCache") MapState<String, SerializableTableSpec> cacheState,
+ @StateId("lastSeen") MapState<String, Long> lastSeenState,
+ OutputReceiver<Map<String, SerializableTableSpec>> out) {
+ long now = clock.currentTimeMillis();
+ KV<String, @Nullable SerializableTableSpec> kv = element.getValue();
+ String tableId = kv.getKey();
+ @Nullable SerializableTableSpec newSpec = kv.getValue();
+
+ if (newSpec != null) {
+ ReadableState<SerializableTableSpec> existingState = cacheState.get(tableId);
+ SerializableTableSpec existingSpec = existingState != null ? existingState.read() : null;
+ if (existingSpec == null || isNewer(newSpec, existingSpec)) {
+ cacheState.put(tableId, newSpec);
+ }
+ lastSeenState.put(tableId, now);
+ } else {
+ // Explicit missing table signal: immediately invalidate any cached entry
+ cacheState.remove(tableId);
+ lastSeenState.remove(tableId);
+ }
+
+ Map<String, Long> lastSeenMap = new HashMap<>();
+ for (Map.Entry<String, Long> entry : lastSeenState.entries().read()) {
+ lastSeenMap.put(entry.getKey(), entry.getValue());
+ }
+ if (newSpec != null) {
+ lastSeenMap.put(tableId, now);
+ }
Review Comment:
nit: unneeded since we're already doing `lastSeenState.put(tableId, now);` above?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.state.MapState;
+import org.apache.beam.sdk.state.ReadableState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Deduplicate;
+import org.apache.beam.sdk.transforms.Distinct;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.StateId;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.Sample;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.checkerframework.checker.nullness.qual.Nullable;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A driver transform that extracts table identifiers from incoming {@link Row}s, deduplicates them
+ * per window, optionally bounds the cache size up to {@code maximumCacheSize} (batch pipelines
+ * only), loads their declarative metadata from the Iceberg catalog, and emits {@link KV} pairs of
+ * table identifier strings to {@link SerializableTableSpec} (or {@code null} if the table does not
+ * exist or fails to load). This is intended to be used in Beam pipelines that may utilize a large
+ * number of workers to handle Iceberg writes, where having every worker thread query for table
+ * metadata results in an excessive amount of requests and a high level of redundancy.
+ *
+ * <p>Can also be materialized into a broadcasted {@link PCollectionView} via {@link
+ * #asView(IcebergCatalogConfig, DynamicDestinations)}. By default, the cache size is uncapped. If
+ * {@code maximumCacheSize} is configured and the number of distinct tables in a window exceeds it,
+ * up to {@code maximumCacheSize} tables are sampled into the broadcasted view, while remaining
+ * destinations fall back to worker-local catalog loading. Note that {@code maximumCacheSize} is
+ * currently supported for bounded batch pipelines only.
+ *
+ * <p>For unbounded streaming pipelines in {@link GlobalWindows}, {@link Deduplicate} is used to
+ * deduplicate table identifiers over the configured {@code refreshInterval} (defaulting to {@link
+ * #DEFAULT_REFRESH_INTERVAL}), allowing periodic refresh of table metadata when schemas evolve.
+ * Missing table signals ({@code null} specs) trigger side-input view materialization without
+ * caching the missing tables, ensuring downstream consumers are never blocked waiting for the side
+ * input.
+ */
+@Internal
+@AutoValue
+public abstract class TableMetadataDriver
+ extends PTransform<PCollection<Row>, PCollection<KV<String, @Nullable SerializableTableSpec>>> {
+
+ public static final Duration DEFAULT_REFRESH_INTERVAL = Duration.standardMinutes(5);
+ public static final int DEFAULT_POLLING_BUCKETS = 1;
+
+ @FunctionalInterface
+ public interface Clock extends Serializable {
+ long currentTimeMillis();
+ }
+
+ public abstract IcebergCatalogConfig getCatalogConfig();
+
+ public abstract DynamicDestinations getDynamicDestinations();
+
+ public abstract @Nullable Integer getMaximumCacheSize();
+
+ public abstract @Nullable Duration getRefreshInterval();
+
+ /**
+ * Returns the number of parallel buckets/workers used to query the Iceberg catalog, or {@code
+ * null} for default.
+ */
+ public abstract @Nullable Integer getPollingBuckets();
+
+ public static Builder builder() {
+ return new AutoValue_TableMetadataDriver.Builder();
+ }
+
+ public abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setCatalogConfig(IcebergCatalogConfig catalogConfig);
+
+ public abstract Builder setDynamicDestinations(DynamicDestinations dynamicDestinations);
+
+ public abstract Builder setMaximumCacheSize(@Nullable Integer maximumCacheSize);
+
+ public abstract Builder setRefreshInterval(@Nullable Duration refreshInterval);
+
+ /**
+ * Sets the number of parallel buckets (worker tasks) used to query the Iceberg catalog.
+ *
+ * <p>Defaults to {@link #DEFAULT_POLLING_BUCKETS} (1), which serializes all catalog lookups to
+ * avoid overwhelming catalog metastores (e.g. Hive Metastore, REST catalog). For pipelines
+ * writing to a large number of distinct dynamic tables (e.g. hundreds of tables per window),
+ * consider increasing this value (e.g. 5–10) to parallelize catalog lookups while still
+ * bounding load.
+ */
+ public abstract Builder setPollingBuckets(@Nullable Integer pollingBuckets);
+
+ abstract TableMetadataDriver autoBuild();
+
+ public TableMetadataDriver build() {
+ TableMetadataDriver driver = autoBuild();
+ Integer maxCacheSize = driver.getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ Preconditions.checkArgument(
+ maxCacheSize > 0, "maximumCacheSize must be greater than 0, got %s", maxCacheSize);
+ }
+ Duration refreshInterval = driver.getRefreshInterval();
+ if (refreshInterval != null) {
+ Preconditions.checkArgument(
+ refreshInterval.isLongerThan(Duration.ZERO),
+ "refreshInterval must be positive, got %s",
+ refreshInterval);
+ }
+ Integer pollingBuckets = driver.getPollingBuckets();
+ if (pollingBuckets != null) {
+ Preconditions.checkArgument(
+ pollingBuckets > 0, "pollingBuckets must be greater than 0, got %s", pollingBuckets);
+ }
+ return driver;
+ }
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} and creates an uncapped {@link PCollectionView}
+ * of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(IcebergCatalogConfig catalogConfig, DynamicDestinations dynamicDestinations) {
+ return asView(catalogConfig, dynamicDestinations, null, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and creates a {@link PCollectionView} of {@link Map} of table identifier strings to {@link
+ * SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, null, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize} limit
+ * and custom {@code refreshInterval}, creating a {@link PCollectionView} of {@link Map} of table
+ * identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval) {
+ return asView(catalogConfig, dynamicDestinations, maximumCacheSize, refreshInterval, null);
+ }
+
+ /**
+ * Helper that applies {@link TableMetadataDriver} with an optional {@code maximumCacheSize}
+ * limit, custom {@code refreshInterval}, and custom {@code pollingBuckets}, creating a {@link
+ * PCollectionView} of {@link Map} of table identifier strings to {@link SerializableTableSpec}.
+ *
+ * @param catalogConfig the catalog configuration used to poll metadata.
+ * @param dynamicDestinations destination strategy extracting table IDs from rows.
+ * @param maximumCacheSize optional maximum distinct tables to poll and broadcast per window (null
+ * for uncapped).
+ * @param refreshInterval optional refresh interval for streaming global window triggers.
+ * @param pollingBuckets optional number of parallel buckets/workers for catalog polling.
+ */
+ public static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>
+ asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Clock clock) {
+ return asView(
+ catalogConfig,
+ dynamicDestinations,
+ maximumCacheSize,
+ refreshInterval,
+ pollingBuckets,
+ null,
+ clock);
+ }
+
+ @VisibleForTesting
+ static PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>> asView(
+ IcebergCatalogConfig catalogConfig,
+ DynamicDestinations dynamicDestinations,
+ @Nullable Integer maximumCacheSize,
+ @Nullable Duration refreshInterval,
+ @Nullable Integer pollingBuckets,
+ @Nullable Duration cacheTtl,
+ @Nullable Clock clock) {
+ return new PTransform<PCollection<Row>, PCollectionView<Map<String, SerializableTableSpec>>>() {
+ @Override
+ public PCollectionView<Map<String, SerializableTableSpec>> expand(PCollection<Row> input) {
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ Duration interval = refreshInterval != null ? refreshInterval : DEFAULT_REFRESH_INTERVAL;
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ input.apply(
+ "GenerateTableMetadata",
+ TableMetadataDriver.builder()
+ .setCatalogConfig(catalogConfig)
+ .setDynamicDestinations(dynamicDestinations)
+ .setMaximumCacheSize(maximumCacheSize)
+ .setRefreshInterval(interval)
+ .setPollingBuckets(pollingBuckets)
+ .build());
+
+ if (isStreaming) {
+ AccumulateTableMetadataMapDoFn accumulateDoFn =
+ clock != null
+ ? new AccumulateTableMetadataMapDoFn(interval, cacheTtl, clock)
+ : new AccumulateTableMetadataMapDoFn(interval, cacheTtl);
+ return specs
+ .apply("KeyForGlobalCache", WithKeys.of((Void) null))
+ .setCoder(KvCoder.of(VoidCoder.of(), specs.getCoder()))
+ .apply("AccumulateCacheMap", ParDo.of(accumulateDoFn))
+ .setCoder(MapCoder.of(StringUtf8Coder.of(), SerializableTableSpec.getCoder()))
+ .apply(
+ "StreamingCacheWindow",
+ Window.<Map<String, SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes())
+ .apply(
+ "CreateMetadataSingletonView",
+ Combine.globally(new MapMergerFn()).asSingletonView());
+ }
+
+ return specs
+ .apply(
+ "FilterValidSpecsForView",
+ Filter.by(
+ (SerializableFunction<KV<String, @Nullable SerializableTableSpec>, Boolean>)
+ kv -> kv.getValue() != null))
+ .apply("CreateTableMetadataView", View.asMap());
+ }
+ };
+ }
+
+ @Override
+ public PCollection<KV<String, @Nullable SerializableTableSpec>> expand(PCollection<Row> input) {
+ PCollection<String> tableIds =
+ input
+ .apply("ExtractTableIds", ParDo.of(new ExtractTableIdsDoFn(getDynamicDestinations())))
+ .setCoder(StringUtf8Coder.of())
+ .apply("MetadataGlobalWindow", Window.into(new GlobalWindows()));
+
+ boolean isStreaming = input.isBounded() == PCollection.IsBounded.UNBOUNDED;
+
+ PCollection<String> distinctTableIds;
+ if (isStreaming) {
+ Duration customInterval = getRefreshInterval();
+ Duration interval =
+ checkNotNull(customInterval != null ? customInterval : DEFAULT_REFRESH_INTERVAL);
+ distinctTableIds =
+ tableIds.apply(
+ "DeduplicateTableIds", Deduplicate.<String>values().withDuration(interval));
+ } else {
+ distinctTableIds = tableIds.apply("DistinctTableIds", Distinct.create());
+ }
+
+ PCollection<String> cachedTableIds;
+ Integer maxCacheSize = getMaximumCacheSize();
+ if (maxCacheSize != null) {
+ if (isStreaming) {
+ throw new UnsupportedOperationException(
+ "maximumCacheSize is currently not supported for unbounded streaming pipelines.");
+ }
+ cachedTableIds = distinctTableIds.apply("CapCacheSize", Sample.any(maxCacheSize));
+ } else {
+ cachedTableIds = distinctTableIds;
+ }
+
+ @Nullable Integer configuredBuckets = getPollingBuckets();
+ int pollingBuckets = configuredBuckets != null ? configuredBuckets : DEFAULT_POLLING_BUCKETS;
+ PCollection<String> pollingTableIds =
+ cachedTableIds.apply(
+ "ReshufflePollingBuckets",
+ Reshuffle.<String>viaRandomKey().withNumBuckets(pollingBuckets));
+
+ PCollection<KV<String, @Nullable SerializableTableSpec>> specs =
+ pollingTableIds
+ .apply("PollTableMetadata", ParDo.of(new CatalogPollingDoFn(getCatalogConfig())))
+ .setCoder(
+ KvCoder.of(
+ StringUtf8Coder.of(), NullableCoder.of(SerializableTableSpec.getCoder())));
+
+ if (isStreaming) {
+ return specs.apply(
+ "ApplyStreamingTrigger",
+ Window.<KV<String, @Nullable SerializableTableSpec>>into(new GlobalWindows())
+ .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+ .discardingFiredPanes());
+ }
+ return specs;
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.addIfNotNull(
+ DisplayData.item("maximumCacheSize", getMaximumCacheSize())
+ .withLabel("Maximum Cache Size"));
+ builder.addIfNotNull(
+ DisplayData.item("refreshInterval", getRefreshInterval())
+ .withLabel("Table Metadata Refresh Interval"));
+ builder.addIfNotNull(
+ DisplayData.item("pollingBuckets", getPollingBuckets())
+ .withLabel("Catalog Polling Buckets"));
+ }
+
+ static class ExtractTableIdsDoFn extends DoFn<Row, String> {
+ private final DynamicDestinations dynamicDestinations;
+
+ ExtractTableIdsDoFn(DynamicDestinations dynamicDestinations) {
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element Row element,
+ BoundedWindow window,
+ PaneInfo paneInfo,
+ @Timestamp Instant timestamp,
+ OutputReceiver<String> out) {
+ String tableIdentifier =
+ dynamicDestinations.getTableStringIdentifier(
+ ValueInSingleWindow.of(element, timestamp, window, paneInfo));
+ if (tableIdentifier != null && !tableIdentifier.trim().isEmpty()) {
+ out.output(tableIdentifier.trim());
+ }
+ }
+ }
+
+ static class CatalogPollingDoFn
+ extends DoFn<String, KV<String, @Nullable SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(CatalogPollingDoFn.class);
+ private static final Counter TABLES_POLLED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesPolled");
+ private static final Counter TABLES_SKIPPED_MISSING_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSkippedMissing");
+ private static final Counter TABLES_PARSE_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesParseFailed");
+ private static final Counter TABLES_SPEC_CREATION_FAILED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesSpecCreationFailed");
+
+ private final IcebergCatalogConfig catalogConfig;
+
+ CatalogPollingDoFn(IcebergCatalogConfig catalogConfig) {
+ this.catalogConfig = catalogConfig;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element String tableIdString,
+ OutputReceiver<KV<String, @Nullable SerializableTableSpec>> out) {
+ TableIdentifier tableId;
+ try {
+ tableId = IcebergUtils.parseTableIdentifier(tableIdString);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to parse table identifier '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_PARSE_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+
+ Table table;
+ try {
+ table = catalogConfig.catalog().loadTable(tableId);
+ } catch (NoSuchTableException e) {
+ LOG.info(
+ "Table '{}' does not exist in catalog. Emitting empty metadata signal for side-input view.",
+ tableIdString);
+ TABLES_SKIPPED_MISSING_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ SerializableTableSpec spec;
+ try {
+ spec = SerializableTableSpec.fromTable(tableIdString, table);
+ } catch (IllegalArgumentException e) {
+ LOG.warn(
+ "Failed to create SerializableTableSpec for table '{}'. Emitting empty metadata signal for side-input view.",
+ tableIdString,
+ e);
+ TABLES_SPEC_CREATION_FAILED_COUNTER.inc();
+ out.output(KV.of(tableIdString, null));
+ return;
+ }
+ TABLES_POLLED_COUNTER.inc();
+ out.output(KV.of(tableIdString, spec));
+ }
+ }
+
+ static class AccumulateTableMetadataMapDoFn
+ extends DoFn<
+ KV<Void, KV<String, @Nullable SerializableTableSpec>>,
+ Map<String, SerializableTableSpec>> {
+ private static final Logger LOG = LoggerFactory.getLogger(AccumulateTableMetadataMapDoFn.class);
+ private static final Counter TABLES_EVICTED_COUNTER =
+ Metrics.counter(TableMetadataDriver.class, "tablesEvictedUnused");
+ static final int DEFAULT_TTL_MULTIPLIER = 3;
+
+ @StateId("tableCache")
+ private final StateSpec<MapState<String, SerializableTableSpec>> cacheStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), SerializableTableSpec.getCoder());
+
+ @StateId("lastSeen")
+ private final StateSpec<MapState<String, Long>> lastSeenStateSpec =
+ StateSpecs.map(StringUtf8Coder.of(), VarLongCoder.of());
+
+ private final Duration refreshInterval;
+ private final Duration cacheTtl;
+ private final Clock clock;
+
+ AccumulateTableMetadataMapDoFn() {
+ this(DEFAULT_REFRESH_INTERVAL, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval) {
+ this(refreshInterval, null, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval, Clock clock) {
+ this(refreshInterval, null, clock);
+ }
+
+ AccumulateTableMetadataMapDoFn(Duration refreshInterval, @Nullable Duration cacheTtl) {
+ this(refreshInterval, cacheTtl, System::currentTimeMillis);
+ }
+
+ AccumulateTableMetadataMapDoFn(
+ Duration refreshInterval, @Nullable Duration cacheTtl, Clock clock) {
+ this.refreshInterval = refreshInterval != null ? refreshInterval : DEFAULT_REFRESH_INTERVAL;
+ this.cacheTtl =
+ cacheTtl != null ? cacheTtl : this.refreshInterval.multipliedBy(DEFAULT_TTL_MULTIPLIER);
+ this.clock = clock != null ? clock : System::currentTimeMillis;
+ }
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<Void, KV<String, @Nullable SerializableTableSpec>> element,
+ @StateId("tableCache") MapState<String, SerializableTableSpec> cacheState,
+ @StateId("lastSeen") MapState<String, Long> lastSeenState,
+ OutputReceiver<Map<String, SerializableTableSpec>> out) {
+ long now = clock.currentTimeMillis();
+ KV<String, @Nullable SerializableTableSpec> kv = element.getValue();
+ String tableId = kv.getKey();
+ @Nullable SerializableTableSpec newSpec = kv.getValue();
+
+ if (newSpec != null) {
+ ReadableState<SerializableTableSpec> existingState = cacheState.get(tableId);
+ SerializableTableSpec existingSpec = existingState != null ? existingState.read() : null;
+ if (existingSpec == null || isNewer(newSpec, existingSpec)) {
+ cacheState.put(tableId, newSpec);
+ }
+ lastSeenState.put(tableId, now);
+ } else {
+ // Explicit missing table signal: immediately invalidate any cached entry
+ cacheState.remove(tableId);
+ lastSeenState.remove(tableId);
+ }
Review Comment:
I don't think we should remove prematurely. If a tableId already exists in state, let it live out its TTL.
If we end up in this block it's probably due to a bad record or transient error. We shouldn't let that kick out a good entry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Iceberg Table Metadata Driver transform [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey merged PR #39883:
URL: https://github.com/apache/beam/pull/39883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org