You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bryanck (via GitHub)" <gi...@apache.org> on 2023/09/12 21:46:45 UTC
[GitHub] [iceberg] bryanck opened a new pull request, #8555: Flink: provide table supplier in sink for optional table reload
bryanck opened a new pull request, #8555:
URL: https://github.com/apache/iceberg/pull/8555
Currently, when a Flink sink job is started, write operators are serialized with an initial table instance and that instance is used for the lifetime of the job. There are cases where a table needs to be reloaded at runtime by the subtasks. For example, if a REST catalog returns expiring credentials in the table load config response, those credentials cannot later be refreshed and will eventually expire. Also, if a task restarts, the original credentials created with the job will be used and they may have long expired.
Longer term, the ability to reload a table could be used to support schema evolution, though that is not addressed in this PR. The table schema and partition spec from the initial table instance are still used for now here.
This PR updates the initialization of the Flink sink so that a table supplier is used in place of a table instance. This supplier can implement reload/refresh logic as needed. The initial supplier implementation created here simply reloads the table at a given interval from the Iceberg catalog.
This initial supplier implementation puts additional load on the Iceberg catalog, so by default the reload option is turned off. Different table suppliers could be implemented in the future that use a centralized mechanism for obtaining refreshed table instances to cut down on the extra catalog load. Different options were explored, such as using a broadcast or the token delegation manager, but each had limitations or complexities, so it was felt that a better first step would be to introduce the abstraction to allow for table reload with a simple initial implementation of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326410140
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -302,9 +320,10 @@ private void commitPendingResult(
private void deleteCommittedManifests(
List<ManifestFile> manifests, String newFlinkJobId, long checkpointId) {
+ FileIO fileIO = table.io();
Review Comment:
Initially the was to ensure the same file IO was used in the loop. I removed this, now that the table load doesn't happen in the supplier's `get()` method anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332029788
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I'm OK with refreshing every checkpoint, our catalog should be able to handle the load. Ryan early on had the thought of refreshing every n checkpoints which is another option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324498585
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
Review Comment:
this should be a JUnit5 test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324497815
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergStreamWriter.java:
##########
@@ -376,7 +376,7 @@ private OneInputStreamOperatorTestHarness<RowData, WriteResult> createIcebergStr
icebergTable, Maps.newHashMap(), new org.apache.flink.configuration.Configuration());
IcebergStreamWriter<RowData> streamWriter =
- FlinkSink.createStreamWriter(icebergTable, flinkWriteConfig, flinkRowType, null);
+ FlinkSink.createStreamWriter(icebergTable, null, flinkWriteConfig, flinkRowType, null, 0);
Review Comment:
I was thinking whether it would make sense to always default to `0` rather than having to pass it explicitly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326994677
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -581,16 +608,26 @@ static RowType toFlinkRowType(Schema schema, TableSchema requestedSchema) {
static IcebergStreamWriter<RowData> createStreamWriter(
Table table,
+ TableLoader tableLoader,
FlinkWriteConf flinkWriteConf,
RowType flinkRowType,
- List<Integer> equalityFieldIds) {
+ List<Integer> equalityFieldIds,
+ long minReloadIntervalMs) {
Review Comment:
My comment might be lost somewhere, but I think it would be better to receive the TableSupplier here, instead of creating a new "incidentally" same supplier than we do in the FlinkSink.
Also seeing the TableLoader/TableSupplier in one place mad me think how do we separate the functionality of the 2. Do we really need 2 objects providing/reloading tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1719194158
> * This PR was updated so the committer will now only reload table at checkpoints. The committer was already updating the table metadata as part of the Iceberg commit and upon restart. Writers will always use the original table metadata and only use the reloaded table to get an updated FileIO, though the hope is that the writers will used the reloaded metadata in the future.
Thanks!
> * This PR was designed so that a different and more optimal table supplier solution could be plugged in later.
My concern here is that we tie ourselves to a "random" place of time to refresh the metadata. In my opinion we will have very specific events where we need refresh the metadata - namely, when we have a record with unexpected schema. The proposed solution does not point to this direction.
> There were limitations with the options we explored for centralized table reload, such as using the new token delegation framework or using a broadcast (I'd be happy to discuss details if you're interested).
I think @gaborgsomogyi and my self would be happy to discuss this problems, since the token delegation framework was designed to solve the exact same issues (Kerberos/AWS token refresh) in Flink, I think it would be good to solve the token renewal with it, if it is possible.
> We went with this initial solution as a starting point. This feature is disabled by default and marked as experimental, so should only be used in cases where it is known that a job will not overburden the catalog with load requests.
My problem is that having a solution which does not point to the right direction would make future work even harder.
> * This PR was not meant to solve schema evolution problems but rather make a change that will take a step towards that long term goal.
As mentioned above, I think it does not point to the right direction. We need the refresh capability, but we specifically need it with a way to trigger manually. Also refreshing the whole table to get the new credentials seems problematic as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324758226
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +369,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowReloadEnabled() throws Exception {
+ List<Row> rows = Lists.newArrayList(Row.of(1, "hello"), Row.of(2, "world"), Row.of(3, "foo"));
+ DataStream<RowData> dataStream =
+ env.addSource(createBoundedSource(rows), ROW_TYPE_INFO)
+ .map(CONVERTER::toInternal, FlinkCompatibilityUtil.toTypeInfo(SimpleDataUtil.ROW_TYPE));
+
+ FlinkSink.forRowData(dataStream)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .reloadIntervalMs(1000)
Review Comment:
Right, this was more of a sanity test to ensure the sink is created when the reload interval is specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325669935
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +369,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowReloadEnabled() throws Exception {
+ List<Row> rows = Lists.newArrayList(Row.of(1, "hello"), Row.of(2, "world"), Row.of(3, "foo"));
+ DataStream<RowData> dataStream =
+ env.addSource(createBoundedSource(rows), ROW_TYPE_INFO)
+ .map(CONVERTER::toInternal, FlinkCompatibilityUtil.toTypeInfo(SimpleDataUtil.ROW_TYPE));
+
+ FlinkSink.forRowData(dataStream)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .reloadIntervalMs(1000)
Review Comment:
ok thanks for clarifying, that's how I understood the test as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325697318
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -302,9 +320,10 @@ private void commitPendingResult(
private void deleteCommittedManifests(
List<ManifestFile> manifests, String newFlinkJobId, long checkpointId) {
+ FileIO fileIO = table.io();
Review Comment:
nit: any reason this `fileIO` variable is needed here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326415922
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -237,15 +250,20 @@ public void notifyCheckpointComplete(long checkpointId) throws Exception {
// For step#4, we don't need to commit iceberg table again because in step#3 we've committed all
// the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be increasing.
- if (checkpointId > maxCommittedCheckpointId) {
- LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
- commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
- this.maxCommittedCheckpointId = checkpointId;
- } else {
- LOG.info(
- "Skipping committing checkpoint {}. {} is already committed.",
- checkpointId,
- maxCommittedCheckpointId);
+ try {
+ if (checkpointId > maxCommittedCheckpointId) {
+ LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
+ commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
+ this.maxCommittedCheckpointId = checkpointId;
+ } else {
+ LOG.info(
+ "Skipping committing checkpoint {}. {} is already committed.",
+ checkpointId,
+ maxCommittedCheckpointId);
+ }
+ } finally {
+ // reload the table in case new configuration is needed
+ this.table = tableSupplier.get();
Review Comment:
Yes, I made this change as part of the update to make refresh more explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327019384
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+public class ReloadingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReloadingTableSupplier.class);
+
+ static final double JITTER_PCT = 0.01;
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final long reloadIntervalMs;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ public ReloadingTableSupplier(
+ Table initialTable, TableLoader tableLoader, long reloadIntervalMs) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(reloadIntervalMs > 0, "reloadIntervalMs must be > 0");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.reloadIntervalMs = reloadIntervalMs;
+ this.nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ }
+
+ @VisibleForTesting
+ long calcNextReloadTimeMs(long now) {
+ long jitter = Math.round(Math.random() * JITTER_PCT * reloadIntervalMs);
+ return now + reloadIntervalMs + jitter;
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ table = initialTable;
+ }
+
+ if (System.currentTimeMillis() > nextReloadTimeMs) {
+ if (!tableLoader.isOpen()) {
+ tableLoader.open();
+ }
+ table = tableLoader.loadTable();
+ nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ LOG.info("Table {} reloaded, next load time is at {}", table.name(), nextReloadTimeMs);
Review Comment:
My feeling was that refresh would happen fairly often, and spamming the log with these messages would be unnecessary. What is the timeline you are planning to use?
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -108,7 +130,10 @@ public RowDataTaskWriterFactory(
@Override
public void initialize(int taskId, int attemptId) {
this.outputFileFactory =
- OutputFileFactory.builderFor(table, taskId, attemptId).format(format).build();
+ OutputFileFactory.builderFor(initTable, taskId, attemptId)
+ .format(format)
+ .ioSupplier(() -> tableSupplier.get().io())
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1323981398
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -207,7 +217,7 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
long checkpointId = context.getCheckpointId();
LOG.info(
"Start to flush snapshot state to state backend, table: {}, checkpointId: {}",
- table,
+ tableSupplier.get(),
Review Comment:
I think it would be better to get the `table` once per commit, and keep using that table during the given commit operation. Fetching the table every time could cause concurrency issues/surprises which are hard to track/reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324325374
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -53,6 +55,8 @@ public class RowDataTaskWriterFactory implements TaskWriterFactory<RowData> {
private transient OutputFileFactory outputFileFactory;
+ /** @deprecated Use constructor with table supplier instead. */
Review Comment:
should this also say `...will be removed in 1.X.0...` as we do for other deprecated code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324499734
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
+
+public class TestReloadingTableSupplier {
+
+ @Test
+ public void testCheckArguments() {
+ Table initialTable = mock(Table.class);
+
+ Table loadedTable = mock(Table.class);
+ TableLoader tableLoader = mock(TableLoader.class);
+ when(tableLoader.loadTable()).thenReturn(loadedTable);
+
+ new ReloadingTableSupplier(initialTable, tableLoader, 100);
+
+ assertThatThrownBy(() -> new ReloadingTableSupplier(initialTable, tableLoader, 0))
+ .isInstanceOf(IllegalArgumentException.class);
Review Comment:
all of these should check for the underlying message, so that we're certain that the right error check is hit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330946559
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
Review Comment:
should we change `Table ...` to `SerializableTable ...`? technically, only the later is correct
##########
flink/v1.17/build.gradle:
##########
@@ -125,6 +125,10 @@ project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}") {
testImplementation libs.awaitility
}
+
+ test {
Review Comment:
why is this needed?
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
if we make this class self-refreshing in the `get()` method, this method can be private. that would match the name `Caching` probably better
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java:
##########
@@ -54,16 +59,28 @@ public void open() {
this.attemptId = getRuntimeContext().getAttemptNumber();
this.writerMetrics = new IcebergStreamWriterMetrics(super.metrics, fullTableName);
- // Initialize the task writer factory.
+ // Initialize the task writer factory before refreshing the table so that the initial
+ // schema and partition spec are used.
this.taskWriterFactory.initialize(subTaskId, attemptId);
+ // Refresh the table if needed.
+ if (tableSupplier instanceof CachingTableSupplier) {
Review Comment:
once we made `CachingTableSupplier` self refreshing, we won't need this anymore
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +370,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowWithWriteTableLoader() throws Exception {
Review Comment:
this method name needs to be updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330963619
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
Review Comment:
I think it is safer to make it explicit, I will make this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1726944571
> thanks. I think we are very close now. left a few minor comments
Thanks for your patience. I made the changes except refreshing in the supplier `get()`. I tend to agree w/ Peter that it should be refreshed only during the checkpoint, but I'm happy to change it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332041834
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
wondering if stale `initialTable` object can be used after restoring from checkpoint? is it correct to actually store the `initialTable`? should it always load the table on first get?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332408911
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
That makes sense to me, and for now, moving the refresh back into the supplier `get()` where it was originally should be fine, as we're only using the refreshed table for the FileIO anyway right now. I'll make that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329546497
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ public void setTable(Table table) {
Review Comment:
instead of a setter, we can add a new constructor that takes in `Supplier<Table>` like other classes. the original constructor can just forward to this constructor with an identity supplier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330954127
##########
flink/v1.17/build.gradle:
##########
@@ -125,6 +125,10 @@ project(":iceberg-flink:iceberg-flink-${flinkMajorVersion}") {
testImplementation libs.awaitility
}
+
+ test {
Review Comment:
@nastra noted that for any new tests we should use JUnit 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324694350
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -207,7 +217,7 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
long checkpointId = context.getCheckpointId();
LOG.info(
"Start to flush snapshot state to state backend, table: {}, checkpointId: {}",
- table,
+ tableSupplier.get(),
Review Comment:
I updated the committer so the table reload happens after the commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332036600
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I am leaning toward purely time based refresh interval if there is no requirements for synchronizing the switch point at checkpoint barrier, which is always a bit tricky to guarantee. We might just need to enhance `IcebergFilesCommitter` to handle partition spec evolution. But that is for future scope of supporting schema and partition spec evolution in the sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331879227
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
The interval is a minimum threshold (it was named `minRefreshInterval` originally). We don't necessarily want to refresh at every checkpoint. Though we are doing that in the Kafka Connect sink and haven't had issues, but that probably is not being used at the same scale as Flink jobs.
It should be fine to refresh the schema, spec, and FileIO all together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329439775
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +268,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the table loader used to refresh the table instance in {@link IcebergStreamWriter}. If
+ * not specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param newTableLoader the table loader to use to refresh the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder writeTableLoader(TableLoader newTableLoader) {
Review Comment:
or this can be `tableRefreshInternal(Duration refreshInterval)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330958770
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java:
##########
@@ -54,16 +59,28 @@ public void open() {
this.attemptId = getRuntimeContext().getAttemptNumber();
this.writerMetrics = new IcebergStreamWriterMetrics(super.metrics, fullTableName);
- // Initialize the task writer factory.
+ // Initialize the task writer factory before refreshing the table so that the initial
+ // schema and partition spec are used.
this.taskWriterFactory.initialize(subTaskId, attemptId);
+ // Refresh the table if needed.
+ if (tableSupplier instanceof CachingTableSupplier) {
Review Comment:
As noted above, refreshing in the `get()` could lead to refreshes happening at indeterminate times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1719726330
@pvary
> My concern here is that we tie ourselves to a "random" place of time to refresh the metadata. In my opinion we will have very specific events where we need refresh the metadata - namely, when we have a record with unexpected schema. The proposed solution does not point to this direction.
I wouldn't say it is random. In the committer it happens after a commit. In the writer, it happens when a writer is initialized.
>
> I think @gaborgsomogyi and my self would be happy to discuss this problems, since the token delegation framework was designed to solve the exact same issues (Kerberos/AWS token refresh) in Flink, I think it would be good to solve the token renewal with it, if it is possible.
>
I have a working prototype of refreshing the table using the delegation manager. The main complication is that the manager is iniailized as part of the JM/TM initialization, not application initialization. This requires configuring the catalog for the delegation manager independently and introducing a separate configuration. It also required the application run in the same classloader as the JM or Tm. If those can be addressed then it should be easy to plug in a table supplier that reads from that instead of the catalog.
> My problem is that having a solution which does not point to the right direction would make future work even harder.
> As mentioned above, I think it does not point to the right direction. We need the refresh capability, but we specifically need it with a way to trigger manually. Also refreshing the whole table to get the new credentials seems problematic as well.
Which part of this do you feed is not going in the right direction? The main change is introducing an abstraction that allows a table to be refreshed, which is the first step to any solution. The actual reloading table supplier is a very small part if this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325700858
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergStreamWriter.java:
##########
@@ -376,7 +376,7 @@ private OneInputStreamOperatorTestHarness<RowData, WriteResult> createIcebergStr
icebergTable, Maps.newHashMap(), new org.apache.flink.configuration.Configuration());
IcebergStreamWriter<RowData> streamWriter =
- FlinkSink.createStreamWriter(icebergTable, flinkWriteConfig, flinkRowType, null);
+ FlinkSink.createStreamWriter(icebergTable, null, flinkWriteConfig, flinkRowType, null, 0);
Review Comment:
I think it's ok to go with what you have. It was just a random thought I had while reviewing, but wasn't sure whether it's worth doing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326414370
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
`TableLoader` is specifically for loading a table from a catalog (or HadoopTables). A `TableSupplier` implementation might get the table from someplace else, like the delegation framework or a broadcast (the initial TableSupplier implementation uses a TableLoader).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722157440
> @bryanck I have 2 questions to understand the problem here
>
> 1. how is this handled in Spark? I assume credentials could be valid for a few hours and a Spark job can run for 10+ hours. so this problem also applies to Spark.
> 2. can the credentials refresh capability be baked into `FileIO`?
Hi Steven, because the table is refreshed before each query in Spark, it would only impact queries running longer than credential expiration, and that hasn't been a issue for us specifically. It might be something that needs to be addressed at some point.
We considered baking something into the FileIO. We felt that adding table refresh capability in the writers is already something that has been requested for a while, can provide credential and other configuration updates, and the REST API is already spec'd and available.
Implemenation-wise, a centralized solution for refreshing the table metadata is ideal. The implementation here, where the subtasks perform the refresh, still is works for many cases. If it is a point of contention, we can remove the implementation (ReloadingTableSupplier) but keep the ability to set one.
Implementation-wise, a FileIO-specific API for credential refresh will still add load to an Iceberg REST implementation. Table load with the REST API and refs-only snapshot mode can be fairly efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330352069
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ public void setTable(Table table) {
Review Comment:
I made this package private, but otherwise left it as-is. The initial table needs to be serialized with the instance to retain the same behavior for schema and spec as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1725946023
@stevenzwu @pvary I addressed the feedback, a couple of items I left as-is and left a comment in case you wanted to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330957839
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
Review Comment:
I was debating that, though that isn't really done elsewhere in the Iceberg codebase I don't think. For the supplier we need it, otherwise we'd need to typecast that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330961931
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +370,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowWithWriteTableLoader() throws Exception {
Review Comment:
Thanks, I missed this, this is updated now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332066557
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
yes, I also checked the refresh call on open method. but refresh always check the time threshold. so before the time threshold elapsed, stale `initialTable` is use, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331887793
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
once we use time threshold, then we won't refresh at checkpoint boundary anymore. those 2 are conflicting. we probably have to stay with only one (not both)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1323987118
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -108,7 +130,10 @@ public RowDataTaskWriterFactory(
@Override
public void initialize(int taskId, int attemptId) {
this.outputFileFactory =
- OutputFileFactory.builderFor(table, taskId, attemptId).format(format).build();
+ OutputFileFactory.builderFor(initTable, taskId, attemptId)
+ .format(format)
+ .ioSupplier(() -> tableSupplier.get().io())
Review Comment:
Could it be that the table io is changing during the lifetime of the table? Do we plan to accommodate those changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326414370
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
`TableLoader` is specifically for loading a table from a catalog. A `TableSupplier` implementation might get the table from someplace else, like the delegation framework or a broadcast (the initial TableSupplier implementation uses a TableLoader).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324734942
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+public class ReloadingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReloadingTableSupplier.class);
+
+ static final double JITTER_PCT = 0.01;
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final long reloadIntervalMs;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ public ReloadingTableSupplier(
+ Table initialTable, TableLoader tableLoader, long reloadIntervalMs) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(reloadIntervalMs > 0, "reloadIntervalMs must be > 0");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.reloadIntervalMs = reloadIntervalMs;
+ this.nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ }
+
+ @VisibleForTesting
+ long calcNextReloadTimeMs(long now) {
+ long jitter = Math.round(Math.random() * JITTER_PCT * reloadIntervalMs);
+ return now + reloadIntervalMs + jitter;
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ table = initialTable;
+ }
+
+ if (System.currentTimeMillis() > nextReloadTimeMs) {
+ if (!tableLoader.isOpen()) {
+ tableLoader.open();
+ }
+ table = tableLoader.loadTable();
+ nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ LOG.info("Table {} reloaded, next load time is at {}", table.name(), nextReloadTimeMs);
Review Comment:
I added the date formatting. I feel like info level is appropriate, this is a fairly important operation that occurs infrequently so shouldn't cause log spam. Was there a reason you'd prefer it at debug?
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -53,6 +55,8 @@ public class RowDataTaskWriterFactory implements TaskWriterFactory<RowData> {
private transient OutputFileFactory outputFileFactory;
+ /** @deprecated Use constructor with table supplier instead. */
Review Comment:
I added this to the comment (with the hope this can make it into 1.4.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327028230
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -131,13 +134,15 @@ class IcebergFilesCommitter extends AbstractStreamOperator<Void>
Map<String, String> snapshotProperties,
Integer workerPoolSize,
String branch,
- PartitionSpec spec) {
+ PartitionSpec spec,
+ long minReloadIntervalMs) {
this.tableLoader = tableLoader;
this.replacePartitions = replacePartitions;
this.snapshotProperties = snapshotProperties;
this.workerPoolSize = workerPoolSize;
this.branch = branch;
this.spec = spec;
+ this.minReloadIntervalMs = minReloadIntervalMs;
Review Comment:
Oh.. I found my previous comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1730238929
Thanks @stevenzwu and @pvary for taking the time on this, and for the feedback and guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332287112
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
> but for schema evolution I think you'd want it on checkpoint boundaries.
This is the assumption that I like us to challenge. It was also my initial assumption too.
For partition spec evolution, we need to enhance `IcebergFilesCommitter` as a single manifest file is tied to one partition spec. if we have data files from two different partition specs (during transition period), committer can separate them out and commit in separate transactions.
For schema evolution, I am not sure if we need to do anything special if data files have 2 different schemas (during transition period).
In the future coordinator broadcast approach, it can be more complicated to apply the switch at checkpoint boundary and probably more difficult to guarantee it. We can (1) either add the complexity in the committer operator to enhance the handling of partition spec evolution (2) or add the complexity in the coordinator broadcast implementation. I am wondering if the former is a bit cleaner and easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1333293237
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I checked in an alternative solution, where the table is refreshed only before creating a new task writer. This keeps refresh logic out of `IcebergStreamWriter`, but prevents the table from being updated unintentionally when getting the table reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324718894
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java:
##########
@@ -70,7 +65,10 @@ private String generatePath(long checkpointId) {
}
OutputFile create(long checkpointId) {
- String flinkManifestDir = props.get(FLINK_MANIFEST_LOCATION);
+ Table table = tableSupplier.get();
+ TableOperations ops = ((HasTableOperations) table).operations();
+
+ String flinkManifestDir = table.properties().get(FLINK_MANIFEST_LOCATION);
Review Comment:
I reverted this so the properties are stored as a member variable as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324485598
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -32,6 +32,7 @@
import java.util.Map;
import java.util.Set;
import java.util.function.Function;
+import org.apache.calcite.linq4j.function.Experimental;
Review Comment:
should this be `org.apache.flink.annotation.Experimental`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324492798
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+public class ReloadingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReloadingTableSupplier.class);
+
+ static final double JITTER_PCT = 0.01;
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final long reloadIntervalMs;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ public ReloadingTableSupplier(
+ Table initialTable, TableLoader tableLoader, long reloadIntervalMs) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(reloadIntervalMs > 0, "reloadIntervalMs must be > 0");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.reloadIntervalMs = reloadIntervalMs;
+ this.nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ }
+
+ @VisibleForTesting
+ long calcNextReloadTimeMs(long now) {
+ long jitter = Math.round(Math.random() * JITTER_PCT * reloadIntervalMs);
+ return now + reloadIntervalMs + jitter;
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ table = initialTable;
+ }
+
+ if (System.currentTimeMillis() > nextReloadTimeMs) {
+ if (!tableLoader.isOpen()) {
+ tableLoader.open();
+ }
+ table = tableLoader.loadTable();
+ nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ LOG.info("Table {} reloaded, next load time is at {}", table.name(), nextReloadTimeMs);
Review Comment:
maybe for printing we could convert this to an Instant (using `DateTimeUtil`), so that it's actually readable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324843406
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergStreamWriter.java:
##########
@@ -376,7 +376,7 @@ private OneInputStreamOperatorTestHarness<RowData, WriteResult> createIcebergStr
icebergTable, Maps.newHashMap(), new org.apache.flink.configuration.Configuration());
IcebergStreamWriter<RowData> streamWriter =
- FlinkSink.createStreamWriter(icebergTable, flinkWriteConfig, flinkRowType, null);
+ FlinkSink.createStreamWriter(icebergTable, null, flinkWriteConfig, flinkRowType, null, 0);
Review Comment:
That would only be used by the tests, as it is always specified otherwise. Do you still feel we should add that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326995506
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -131,13 +134,15 @@ class IcebergFilesCommitter extends AbstractStreamOperator<Void>
Map<String, String> snapshotProperties,
Integer workerPoolSize,
String branch,
- PartitionSpec spec) {
+ PartitionSpec spec,
+ long minReloadIntervalMs) {
this.tableLoader = tableLoader;
this.replacePartitions = replacePartitions;
this.snapshotProperties = snapshotProperties;
this.workerPoolSize = workerPoolSize;
this.branch = branch;
this.spec = spec;
+ this.minReloadIntervalMs = minReloadIntervalMs;
Review Comment:
My comment might be lost somewhere, but I think it would be better to receive the TableSupplier here, instead of creating a new "incidentally" same supplier than we do in the FlinkSink.
Also seeing the TableLoader/TableSupplier in one place mad me think how do we separate the functionality of the 2. Do we really need 2 objects providing/reloading tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1730239811
(also thanks to @nastra and @amogh-jahagirdar for the reviews)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330785649
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ void setTable(Table table) {
Review Comment:
I think I can make it work, I'll try reverting back to a `Supplier<Table>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332054022
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java:
##########
@@ -64,4 +65,8 @@ private FlinkWriteOptions() {}
public static final ConfigOption<Integer> WRITE_PARALLELISM =
ConfigOptions.key("write-parallelism").intType().noDefaultValue();
+
+ @Experimental
+ public static final ConfigOption<Long> TABLE_REFRSH_MS =
+ ConfigOptions.key("table-refresh-ms").longType().noDefaultValue();
Review Comment:
I attempted this but it looks like `FlinkConfParser` needs some updates to support duration types. For now, I copied what was being done in `FlinkReadOptions` for a duration type, which makes it a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332058895
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
The table will be refreshed in `IcebergStreamWriter.open()` after the task writer is initialized. Keeping the initial table is a temporary measure, so the initial table schema and partition spec are used by the task writer after a restart, until schema evolution is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330330711
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -237,15 +237,20 @@ public void notifyCheckpointComplete(long checkpointId) throws Exception {
// For step#4, we don't need to commit iceberg table again because in step#3 we've committed all
// the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be increasing.
- if (checkpointId > maxCommittedCheckpointId) {
- LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
- commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
- this.maxCommittedCheckpointId = checkpointId;
- } else {
- LOG.info(
- "Skipping committing checkpoint {}. {} is already committed.",
- checkpointId,
- maxCommittedCheckpointId);
+ try {
+ if (checkpointId > maxCommittedCheckpointId) {
+ LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
+ commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
+ this.maxCommittedCheckpointId = checkpointId;
+ } else {
+ LOG.info(
+ "Skipping committing checkpoint {}. {} is already committed.",
+ checkpointId,
+ maxCommittedCheckpointId);
+ }
+ } finally {
Review Comment:
Yes, thanks, I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330329488
##########
core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java:
##########
@@ -158,12 +161,24 @@ public Builder suffix(String newSuffix) {
return this;
}
+ public Builder ioSupplier(Supplier<FileIO> newIoSupplier) {
Review Comment:
Thanks, I added this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332413725
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I checked in that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1323984739
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+public class ReloadingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReloadingTableSupplier.class);
+
+ static final double JITTER_PCT = 0.01;
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final long reloadIntervalMs;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ public ReloadingTableSupplier(
+ Table initialTable, TableLoader tableLoader, long reloadIntervalMs) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(reloadIntervalMs > 0, "reloadIntervalMs must be > 0");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.reloadIntervalMs = reloadIntervalMs;
+ this.nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ }
+
+ @VisibleForTesting
+ long calcNextReloadTimeMs(long now) {
+ long jitter = Math.round(Math.random() * JITTER_PCT * reloadIntervalMs);
+ return now + reloadIntervalMs + jitter;
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ table = initialTable;
+ }
+
+ if (System.currentTimeMillis() > nextReloadTimeMs) {
+ if (!tableLoader.isOpen()) {
+ tableLoader.open();
+ }
+ table = tableLoader.loadTable();
+ nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ LOG.info("Table {} reloaded, next load time is at {}", table.name(), nextReloadTimeMs);
Review Comment:
Should this be on debug level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325727176
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -117,6 +145,8 @@ public TaskWriter<RowData> create() {
outputFileFactory,
"The outputFileFactory shouldn't be null if we have invoked the initialize().");
+ FileIO io = tableSupplier.get().io();
Review Comment:
Why is this needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1719843947
@pvary I made the change so the refresh is more explicit now, the reload requires a call to `refreshTable()` now, rather than automatically refreshing in the `get()` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326415922
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -237,15 +250,20 @@ public void notifyCheckpointComplete(long checkpointId) throws Exception {
// For step#4, we don't need to commit iceberg table again because in step#3 we've committed all
// the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be increasing.
- if (checkpointId > maxCommittedCheckpointId) {
- LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
- commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
- this.maxCommittedCheckpointId = checkpointId;
- } else {
- LOG.info(
- "Skipping committing checkpoint {}. {} is already committed.",
- checkpointId,
- maxCommittedCheckpointId);
+ try {
+ if (checkpointId > maxCommittedCheckpointId) {
+ LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
+ commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
+ this.maxCommittedCheckpointId = checkpointId;
+ } else {
+ LOG.info(
+ "Skipping committing checkpoint {}. {} is already committed.",
+ checkpointId,
+ maxCommittedCheckpointId);
+ }
+ } finally {
+ // reload the table in case new configuration is needed
+ this.table = tableSupplier.get();
Review Comment:
Yes, I made a few changes as part of the update to make refresh more explicit, and included this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327846264
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
I removed the table supplier in the committer, it only uses a table loader now. In the writer, the supplier is owned by the task writer factory, not the operator, but the operator is responsible for triggering the refresh. The current TableLoader interface doesn't provide a good way to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330330368
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -142,4 +160,35 @@ public String toString() {
.toString();
}
}
+
+ class NoOpTableLoader implements TableLoader {
Review Comment:
Yes, both of these are better names, I changed it to `ImmutableTableLoader`.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +268,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the table loader used to refresh the table instance in {@link IcebergStreamWriter}. If
+ * not specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param newTableLoader the table loader to use to refresh the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder writeTableLoader(TableLoader newTableLoader) {
Review Comment:
I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332130088
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
>The time of the initial table load is also serialized, so if it is stale then the refresh will update it.
ah. I missed this point earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329543555
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/CachingTableLoader.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+@Experimental
+public class CachingTableLoader implements TableLoader {
Review Comment:
after looking at other classes using `Supplier<Table>`, I think this could be called `RefreshableTableSupplier` that implements `Supplier<Table>`. This class should be non public or annotated with `@Internal`. only `FlinkSink` uses it depends on the `tableRefreshInternal(Duration refreshInterval)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330358064
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java:
##########
@@ -57,13 +61,24 @@ public void open() {
// Initialize the task writer factory.
this.taskWriterFactory.initialize(subTaskId, attemptId);
+ // Refresh the table if needed.
+ this.tableLoader.open();
+ if (this.taskWriterFactory instanceof RowDataTaskWriterFactory) {
Review Comment:
The instance check here is to ensure the typecast is safe. The table loader passed in will either be `ImmutableTableLoader` or `CachingTableLoader` so calling `loadTable` should behave appropriately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332018527
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
exactly, that's why I think we should stick with one: either just refresh on checkpoint boundary or refresh with time boundary independently by tasks.
if we stick with checkpoint boundary, then there is no need for the `Duration` arg for `FlinkSink$Builder#tableRefreshInterval(...)`, a simple boolean is fine. I am also thinking about future implementation from coordinator broadcast. Coordinator can send/broadcast the refreshed table object first, tasks will only apply the switch at next checkpoint boundary. Whether coordinator refresh at time or checkpoint boundary will be same with what we need to decide here.
For the scope of `FileIO` or `credentials` refresh, async/non-synchronized switch (time threshold based) would be fine. For schema/partition evolution, is it necessary to switch at the checkpoint/flush boundary across tasks? It may not be necessary. For partition spec evolution, we can enhance the `IcebergFilesCommitter` to group data and delete files by partition specs and commit one transaction per partition spec (during transition/switch periods). For schema evolution, it seems that we don't need to do anything special if writer tasks switched at different time (or checkpoint cycles). It is ok to commit data files with different schemas in one transaction.
@pvary @danielcweeks @rdblue what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332065136
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java:
##########
@@ -64,4 +65,8 @@ private FlinkWriteOptions() {}
public static final ConfigOption<Integer> WRITE_PARALLELISM =
ConfigOptions.key("write-parallelism").intType().noDefaultValue();
+
+ @Experimental
+ public static final ConfigOption<Long> TABLE_REFRSH_MS =
+ ConfigOptions.key("table-refresh-ms").longType().noDefaultValue();
Review Comment:
it should be `durationType` not `stringType`. we need to add the duration type support in `FlinkConfParser`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332068567
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java:
##########
@@ -64,4 +65,8 @@ private FlinkWriteOptions() {}
public static final ConfigOption<Integer> WRITE_PARALLELISM =
ConfigOptions.key("write-parallelism").intType().noDefaultValue();
+
+ @Experimental
+ public static final ConfigOption<Long> TABLE_REFRSH_MS =
+ ConfigOptions.key("table-refresh-ms").longType().noDefaultValue();
Review Comment:
OK I'll make those updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331868368
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
got it. I am fine with that reasoning.
but with that, should `tableRefreshInterval(Duration)` take a simple boolean arg? Do we always want to tie the table refresh interval with checkpoint interval? do we Flink jobs checkpoint pretty frequently (like seconds)?
I am also thinking about in the future when we want to add coordinator broadcast and schema evolution support. it is always ok to refresh io along with schema/partition spec, right? I think yes. since we are refreshing table, those all come together, it is no harm to refresh them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332481528
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
@stevenzwu: How dynamically refreshing the table (partitioning schema) between checkpoints would effect the shuffling of the writers? I think, beside the committer change, the statistics part is needed to be tweaked too to supply the 2nd type of schema evolution. Also data with dynamically changing schema without a schema registry would not be performant because of serialization issues. So I lean to supporting only the 1st version of the schema evolution ATM.
That said, I think if we agree that we are flexible about this part of the change, then we should not block this PR for this decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324656160
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
Review Comment:
The Flink tests are still using `org.junitTest` so I kept it consistent with those.
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
Review Comment:
The Flink tests are still using `org.junit.Test` so I kept it consistent with those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1323989664
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -126,6 +138,7 @@ public void close() throws IOException {
if (catalog instanceof Closeable) {
((Closeable) catalog).close();
}
+ catalog = null;
Review Comment:
Nit: empty line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324320949
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
Review Comment:
should this be `org.apache.flink.annotation.Experimental`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330330040
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/CachingTableLoader.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+@Experimental
+public class CachingTableLoader implements TableLoader {
Review Comment:
I moved this to the `sink` package, made this package private, and removed `@Experimental`. I left it as a `TableLoader`. I originally had it as a `Supplier<Table>` and the feedback was that it was redundant with `TableLoader`.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -38,6 +38,8 @@ public interface TableLoader extends Closeable, Serializable, Cloneable {
void open();
+ boolean isOpen();
Review Comment:
This is needed to prevent the table loader from initializing a catalog more than once unnecessarily. Alternatively it could be checked in the `open()`, but I felt that might have unintended side effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330966165
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
Review Comment:
This is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332032980
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java:
##########
@@ -64,4 +65,8 @@ private FlinkWriteOptions() {}
public static final ConfigOption<Integer> WRITE_PARALLELISM =
ConfigOptions.key("write-parallelism").intType().noDefaultValue();
+
+ @Experimental
+ public static final ConfigOption<Long> TABLE_REFRSH_MS =
+ ConfigOptions.key("table-refresh-ms").longType().noDefaultValue();
Review Comment:
this should be a `durationType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1333250759
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
My main concern before was within a single subtask, rather than across different subtasks. I felt changing the schema or spec while the file is being written could potentially lead to problems down the road, where the refresh isn't done in a deterministic way.
Even using a broadcast or token delegation framework, you could still cache and refresh in the supplier only at given times in the supplier for those. And before the latest change, no public APIs were changed.
Either way, implementing schema evolution is going to require additional changes, so the current PR will be fine to refresh the FileIO and how the refresh occurs can be updated as needed when schema evolution is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332036600
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I am leaning toward purely time based refresh interval if there is no requirements for synchronizing the switch point at checkpoint barrier, which is always a bit tricky to guarantee. We might just need to enhance `IcebergFilesCommitter` to handle partition spec evolution. But that is for future scope of supporting schema and partition spec evolution in the sink. What do others think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331925666
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
because of the time threshold check, some tasks may actually refresh handsome may skip (e.g. due to different init time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332069515
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
The time of the initial table load is also serialized, so if it is stale then the refresh will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1333257617
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
I'd actually prefer to revert back to only refreshing during the checkpoint and having a minimum refresh interval. The subtasks can get out of sync regardless, but it ensures some consistency within a subtask during a checkpoint interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1333167322
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
> That said, I think if we agree that we are flexible about this part of the change, then we should not block this PR for this decision.
I was hoping to get some consensus on this topic/design (not implementation), because it affects the how we should expose this PR in the public API change `tableRefreshInteval(Duration)` vs `enableTableRefresh()`. if we always want to tie the switch at checkpoint boundary, the later boolean version seems more natural because interval is not needed at all. this seems like the only open item left. internal implementation is easy to change. public API needs to be more careful.
@pvary do you see any fundamental problems if tasks make independent timing on switching to new schema (without checkpoint boundary alignment)? I am concerned about checkpoint boundary in the future implementation of coordinator broadcast. what if some broadcast msg came before the checkpoint barrier and some come after? is it really possible for us to fully align all writer tasks perfectly? I felt it is very challenging. the alternative approach is not to rely on it. let the write tasks switch independently. we just enhance the committer to handle potentially multiple partition specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324747538
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java:
##########
@@ -70,7 +65,10 @@ private String generatePath(long checkpointId) {
}
OutputFile create(long checkpointId) {
- String flinkManifestDir = props.get(FLINK_MANIFEST_LOCATION);
+ Table table = tableSupplier.get();
+ TableOperations ops = ((HasTableOperations) table).operations();
+
+ String flinkManifestDir = table.properties().get(FLINK_MANIFEST_LOCATION);
Review Comment:
One note, this will reset the property if the task restarts (same as before).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324734484
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
Review Comment:
Argh, yes, thanks. I updated this also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324794094
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -108,7 +130,10 @@ public RowDataTaskWriterFactory(
@Override
public void initialize(int taskId, int attemptId) {
this.outputFileFactory =
- OutputFileFactory.builderFor(table, taskId, attemptId).format(format).build();
+ OutputFileFactory.builderFor(initTable, taskId, attemptId)
+ .format(format)
+ .ioSupplier(() -> tableSupplier.get().io())
Review Comment:
Using the current table's FileIO to perform file operations should be safe, I think it's actually desirable. For example, if write properties on the table are updated, those will be applied at runtime rather than requiring a restart (only if table reload is explicitly enabled).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1721798291
@pvary I made a small change to the sink builder, so now it takes a `TableSupplier` as an optional parameter instead of the refresh interval. This allows any type of table supplier to be passed in. Also it removes references to `ReloadingTableSupplier` so a user will need to explicitly pass that in to use it. The committer was changed to perform a reload after commit rather than use the supplier to ensure it gets the latest. A user can still specify the reloading supplier but needs to explicitly do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327846264
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
I removed the table supplier in the committer, it only uses a table loader now. In the writer, the supplier is owned by the task writer factory, not the operator, but the operator is responsible for triggering the refresh. The current TableLoader interface doesn't provide a good way to do that. I’ll try moving things around so the supplier interface isn’t needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324554391
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -148,19 +153,24 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
- this.table = tableLoader.loadTable();
- this.committerMetrics = new IcebergFilesCommitterMetrics(super.metrics, table.name());
+ Table initTable = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(initTable, tableLoader, reloadIntervalMs);
+ } else {
+ this.tableSupplier = () -> initTable;
+ }
+ this.committerMetrics = new IcebergFilesCommitterMetrics(super.metrics, initTable.name());
Review Comment:
Nit spacing, newline after the else block
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +369,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowReloadEnabled() throws Exception {
+ List<Row> rows = Lists.newArrayList(Row.of(1, "hello"), Row.of(2, "world"), Row.of(3, "foo"));
+ DataStream<RowData> dataStream =
+ env.addSource(createBoundedSource(rows), ROW_TYPE_INFO)
+ .map(CONVERTER::toInternal, FlinkCompatibilityUtil.toTypeInfo(SimpleDataUtil.ROW_TYPE));
+
+ FlinkSink.forRowData(dataStream)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .reloadIntervalMs(1000)
Review Comment:
If I understand right, internally a reloading table supplier will be created which actually performs the periodic load table. I guess this test is just making sure everything works as expected when this is specified. If there's a simple way for passing some sort of mocked table loader and verifying it gets called multiple times that's a stronger test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324735562
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
+
+public class TestReloadingTableSupplier {
+
+ @Test
+ public void testCheckArguments() {
+ Table initialTable = mock(Table.class);
+
+ Table loadedTable = mock(Table.class);
+ TableLoader tableLoader = mock(TableLoader.class);
+ when(tableLoader.loadTable()).thenReturn(loadedTable);
+
+ new ReloadingTableSupplier(initialTable, tableLoader, 100);
+
+ assertThatThrownBy(() -> new ReloadingTableSupplier(initialTable, tableLoader, 0))
+ .isInstanceOf(IllegalArgumentException.class);
Review Comment:
Thanks, I added these assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324830328
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -148,19 +153,24 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
- this.table = tableLoader.loadTable();
- this.committerMetrics = new IcebergFilesCommitterMetrics(super.metrics, table.name());
+ Table initTable = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(initTable, tableLoader, reloadIntervalMs);
+ } else {
+ this.tableSupplier = () -> initTable;
+ }
+ this.committerMetrics = new IcebergFilesCommitterMetrics(super.metrics, initTable.name());
Review Comment:
I added this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324728263
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
Review Comment:
for new test classes being added we actually want to use JUnit5, as otherwise this will just cause more work later on in the migration process.
You might need to add the below snippet to the build file (if it isn't already defined for the flink project)
```
test {
useJUnitPlatform()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1323983275
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java:
##########
@@ -70,7 +65,10 @@ private String generatePath(long checkpointId) {
}
OutputFile create(long checkpointId) {
- String flinkManifestDir = props.get(FLINK_MANIFEST_LOCATION);
+ Table table = tableSupplier.get();
+ TableOperations ops = ((HasTableOperations) table).operations();
+
+ String flinkManifestDir = table.properties().get(FLINK_MANIFEST_LOCATION);
Review Comment:
What happens, if someone commits a change which sets this property? Do we store absolute path for the manifests? Or we try to recalculate, and potentially fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1720983421
> @pvary
>
> > My concern here is that we tie ourselves to a "random" place of time to refresh the metadata. In my opinion we will have very specific events where we need refresh the metadata - namely, when we have a record with unexpected schema. The proposed solution does not point to this direction.
>
> I wouldn't say it is random. In the committer it happens after a commit. In the writer, it happens when a writer is initialized.
I am fine with the commit - it was something similar already, as we had to reload the table before every commit, we just had to make sure that we do not have unexpected refresh on the table.
The writer did refresh from time-ot-time in the middle of the code before your last change.
> I have a working prototype of refreshing the table using the delegation manager. The main complication is that the manager is iniailized as part of the JM/TM initialization, not application initialization. This requires configuring the catalog for the delegation manager independently and introducing a separate configuration. It also requires the application run in the same classloader as the JM or TM. If those can be addressed then it should be easy to plug in a table supplier that reads from that instead of the catalog.
I think the configuration handling could be solved - I have seen similar issues solved in production
There are 2 possibilities to solve the classloader issues:
- Put the provider/receiver/authentication to a plugin which are using the same JVM. This way the plugin classloader will be used for all of them. I expect the main hurdle here is to redirect the authentication to the plugin
- Put everything on the main Flink classpath - this should be straightforward
> Which part of this do you feed is not going in the right direction? The main change is introducing an abstraction that allows a table to be refreshed, which is the first step to any solution. The actual reloading table supplier is a very small part if this.
This opens up a way to create very bad solutions (calling central components from Tasks - DDoS-ing catalogs), and restricts the future development (we needed only schema refresh for following schema evolution of the tables, and this could be solved by using schemastore solutions. Also the schemastore could be used not only in the Iceberg sink, but in previous steps of the jobs as well, so overall it seems a much better solution than refreshing the tables)
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327668211
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ReloadingTableSupplier.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import org.apache.calcite.linq4j.function.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+public class ReloadingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ReloadingTableSupplier.class);
+
+ static final double JITTER_PCT = 0.01;
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final long reloadIntervalMs;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ public ReloadingTableSupplier(
+ Table initialTable, TableLoader tableLoader, long reloadIntervalMs) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(reloadIntervalMs > 0, "reloadIntervalMs must be > 0");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.reloadIntervalMs = reloadIntervalMs;
+ this.nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ }
+
+ @VisibleForTesting
+ long calcNextReloadTimeMs(long now) {
+ long jitter = Math.round(Math.random() * JITTER_PCT * reloadIntervalMs);
+ return now + reloadIntervalMs + jitter;
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ table = initialTable;
+ }
+
+ if (System.currentTimeMillis() > nextReloadTimeMs) {
+ if (!tableLoader.isOpen()) {
+ tableLoader.open();
+ }
+ table = tableLoader.loadTable();
+ nextReloadTimeMs = calcNextReloadTimeMs(System.currentTimeMillis());
+ LOG.info("Table {} reloaded, next load time is at {}", table.name(), nextReloadTimeMs);
Review Comment:
It depends on the refresh interval set when building the sink, the minimum time is the checkpoint interval which would mean an extra log line per checkpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326995506
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -131,13 +134,15 @@ class IcebergFilesCommitter extends AbstractStreamOperator<Void>
Map<String, String> snapshotProperties,
Integer workerPoolSize,
String branch,
- PartitionSpec spec) {
+ PartitionSpec spec,
+ long minReloadIntervalMs) {
this.tableLoader = tableLoader;
this.replacePartitions = replacePartitions;
this.snapshotProperties = snapshotProperties;
this.workerPoolSize = workerPoolSize;
this.branch = branch;
this.spec = spec;
+ this.minReloadIntervalMs = minReloadIntervalMs;
Review Comment:
My comment might be lost somewhere, but I think it would be better to receive the TableSupplier here, instead of creating a new "incidentally" same supplier than we do in the FlinkSink.
Also seeing the TableLoader/TableSupplier in one place made me think how do we separate the functionality of the 2. Do we really need 2 objects providing/reloading tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327004417
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -207,7 +217,7 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
long checkpointId = context.getCheckpointId();
LOG.info(
"Start to flush snapshot state to state backend, table: {}, checkpointId: {}",
- table,
+ tableSupplier.get(),
Review Comment:
nit: Maybe initTable for consistencies shake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329546497
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ public void setTable(Table table) {
Review Comment:
instead of a setter, we can add a new constructor that takes in `Supplier<Table>` like other classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327011565
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java:
##########
@@ -41,15 +42,13 @@ class ManifestOutputFileFactory {
private final AtomicInteger fileCount = new AtomicInteger(0);
ManifestOutputFileFactory(
- TableOperations ops,
- FileIO io,
+ Supplier<Table> tableSupplier,
Review Comment:
Is this change needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722348662
I made the change to remove `TableSupplier` and reuse `TableLoader` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330782463
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ void setTable(Table table) {
Review Comment:
Currently we still want to serialize the original table instance with this factory at job start, for its schema and spec, until schema evolution is supported.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +269,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the interval for refreshing the table instance in {@link IcebergStreamWriter}. If not
+ * specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param refreshInterval the interval for refreshing the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder tableRefreshInternal(Duration refreshInterval) {
Review Comment:
whoops, thanks for catching that, I fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327041574
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
The first part of the comment is relevant anyway:
- We should not create the supplier in the committer - we should get it from the Sink as an initialization parameter. We should keep the supplier creation in one place.
About the Loader vs. Supplier, I think from the consumer side they have the same functionality - get the table from the source of truth.... Should we use the same interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329395164
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -142,4 +160,35 @@ public String toString() {
.toString();
}
}
+
+ class NoOpTableLoader implements TableLoader {
Review Comment:
is `StaticTableLoader` or `ImmutableTableLoader` more accurate? this is also a cached loader that never expires.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java:
##########
@@ -57,13 +61,24 @@ public void open() {
// Initialize the task writer factory.
this.taskWriterFactory.initialize(subTaskId, attemptId);
+ // Refresh the table if needed.
+ this.tableLoader.open();
+ if (this.taskWriterFactory instanceof RowDataTaskWriterFactory) {
Review Comment:
here we can check if `tableLoader instanceOf CachingTableLoader`?
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +268,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the table loader used to refresh the table instance in {@link IcebergStreamWriter}. If
+ * not specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param newTableLoader the table loader to use to refresh the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder writeTableLoader(TableLoader newTableLoader) {
Review Comment:
wondering if we need a separate `writeTableLoader`? @bryanck you mentioned that committer also can have the same problem. Note that `TableLoader` already has a `clone` method for other reasons/problems. it can be cloned.
if we use the same table loader, I guess we need a way to figure out if the loaded table needs to be refreshed. Should we check the loader class type like `instanceOf CacheTableLoader/RefreshingTableLoader` ? this way we can also have one less public API change.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -237,15 +237,20 @@ public void notifyCheckpointComplete(long checkpointId) throws Exception {
// For step#4, we don't need to commit iceberg table again because in step#3 we've committed all
// the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be increasing.
- if (checkpointId > maxCommittedCheckpointId) {
- LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
- commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
- this.maxCommittedCheckpointId = checkpointId;
- } else {
- LOG.info(
- "Skipping committing checkpoint {}. {} is already committed.",
- checkpointId,
- maxCommittedCheckpointId);
+ try {
+ if (checkpointId > maxCommittedCheckpointId) {
+ LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
+ commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
+ this.maxCommittedCheckpointId = checkpointId;
+ } else {
+ LOG.info(
+ "Skipping committing checkpoint {}. {} is already committed.",
+ checkpointId,
+ maxCommittedCheckpointId);
+ }
+ } finally {
Review Comment:
this doesn't have to be `finally`, right? if commit failed, job will restart anyway.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ public void setTable(Table table) {
Review Comment:
I guess this is the public API change that is not avoidable with the approach in this PR? Maybe mark it as `@Experiemental` too
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -38,6 +38,8 @@ public interface TableLoader extends Closeable, Serializable, Cloneable {
void open();
+ boolean isOpen();
Review Comment:
is this necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1728173197
I made the change to move the refresh interval config to a write property, and removed the config method in `FlinkSink`. This allows the option to be used in Flink SQL also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330812000
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -38,6 +38,8 @@ public interface TableLoader extends Closeable, Serializable, Cloneable {
void open();
+ boolean isOpen();
Review Comment:
I'm still using this API in the new supplier, let me know if you'd still like me to remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1726693845
@stevenzwu I reverted things back to use a `Supplier<Table>` instead of a `TableLoader`. The only open comment is whether to remove `isOpen()` from the `TableLoader` interface, currently it is still being used in the new table supplier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331894471
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
The refresh is called in `prepareSnapshotPreBarrier()` so it should happen at a deterministic time. But It won't happen at every checkpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329547050
##########
core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java:
##########
@@ -158,12 +161,24 @@ public Builder suffix(String newSuffix) {
return this;
}
+ public Builder ioSupplier(Supplier<FileIO> newIoSupplier) {
Review Comment:
nit: add a Javadoc on why this is a supplier for dynamic refreshing capability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331058824
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
@stevenzwu: I am concerned about unintentionally refreshing data during operations where consistency is needed. If we do refresh behind the scenes, we can break the consistency requirement. We either have to comb through the code to fix all these issues, or we keep refresh a different method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332076439
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(
+ SerializableTable initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
Review Comment:
I changed this slightly, so the last load time is stored instead of the next load time, which should be a little bit more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332287112
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
> but for schema evolution I think you'd want it on checkpoint boundaries.
This is the assumption that I like us to challenge. It was also my initial assumption too.
For partition spec evolution, we need to enhance `IcebergFilesCommitter` as a single manifest file is tied to one partition spec. if we have data files from two different partition specs (during transition period), committer can separate them out and commit in separate transactions.
For schema evolution, I am not sure if we need to do anything special if data files have 2 different schemas (during transition period).
In the future coordinator broadcast approach, it can be more complicated to apply the switch at checkpoint boundary and probably more difficult to guarantee it. We can either add the complexity in the committer operator to enhance the handling of partition spec evolution or add the complexity in the coordinator broadcast implementation. I am wondering if the former is a bit cleaner and easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330810025
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ void setTable(Table table) {
Review Comment:
I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1726893429
> @stevenzwu I reverted things back to use a `Supplier<Table>` instead of a `TableLoader`. The only open comment is whether to remove `isOpen()` from the `TableLoader` interface, currently it is still being used in the new table supplier.
thanks. I think we are very close now. left a few minor comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330956549
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
That was how it was originally, @pvary noted it is better to explicitly refresh only during the checkpoint rather than at indeterminate times when `get()` is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1718003552
> I am very concerned about this simplified solution. We should consider:
>
> * Concurrency issues, when the table metadata has been changed between reloads. Maybe we should only refresh the table metadata during checpoints, so we can localize, and handle changes in one place?
> * Load issues, as every TaskManager with a writer task connects to the catalog, and does so periodically. This will become costly very soon. I would prioritize a centralized solution.
> * The current default implementation does not solve the schema evolution problem, since that would mean on-demand table reload instead of periodic reload
>
> If the main goal here is to refresh the credentials, then maybe a proxy solution above the accessor could be a better solution, like the https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java where we get a new Kerberos token when the previous one has expired.
>
> What do you think?
To address your points:
* This PR was updated so the committer will now only reload table at checkpoints. The committer was already updating the table metadata as part of the Iceberg commit and upon restart. Writers will always use the original table metadata and only use the reloaded table to get an updated FileIO, though the hope is that the writers will used the reloaded metadata in the future.
* This PR was designed so that a different and more optimal table supplier solution could be plugged in later. There were limitations with the options we explored for centralized table reload, such as using the new token delegation framework or using a broadcast (I'd be happy to discuss details if you're interested). We went with this initial solution as a starting point. This feature is disabled by default and marked as experimental, so should only be used in cases where it is known that a job will not overburden the catalog with load requests.
* This PR was not meant to solve schema evolution problems but rather make a change that will take a step towards that long term goal.
* The short term goal for our use case was to refresh credentials, so we initially explored adding a FileIO-specific refresh. However, the long term goal is to support table reload in a running Flink job to support schema evolution. We felt that adding in the table refresh capability would be a better step towards reaching that goal, and it could also be used to solve the particular issue we are having.
cc @danielcweeks @rdblue in case you have additional perspective or feedback...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325754483
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
Why do not get the `tableSupplier` from outside? Why do we create it here and in the `FlinkSink` as well?
QQ: What is the difference between the functionality of the `TableLoader` and the `TableSupplier`? Is it just a TableLoader with cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324734173
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -126,6 +138,7 @@ public void close() throws IOException {
if (catalog instanceof Closeable) {
((Closeable) catalog).close();
}
+ catalog = null;
Review Comment:
I added this.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -32,6 +32,7 @@
import java.util.Map;
import java.util.Set;
import java.util.function.Function;
+import org.apache.calcite.linq4j.function.Experimental;
Review Comment:
Oh yes, thanks for catching this. This is updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324828390
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestReloadingTableSupplier.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.awaitility.Awaitility;
+import org.junit.Test;
Review Comment:
OK thanks, I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1325724334
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -237,15 +250,20 @@ public void notifyCheckpointComplete(long checkpointId) throws Exception {
// For step#4, we don't need to commit iceberg table again because in step#3 we've committed all
// the files,
// Besides, we need to maintain the max-committed-checkpoint-id to be increasing.
- if (checkpointId > maxCommittedCheckpointId) {
- LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
- commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
- this.maxCommittedCheckpointId = checkpointId;
- } else {
- LOG.info(
- "Skipping committing checkpoint {}. {} is already committed.",
- checkpointId,
- maxCommittedCheckpointId);
+ try {
+ if (checkpointId > maxCommittedCheckpointId) {
+ LOG.info("Checkpoint {} completed. Attempting commit.", checkpointId);
+ commitUpToCheckpoint(dataFilesPerCheckpoint, flinkJobId, operatorUniqueId, checkpointId);
+ this.maxCommittedCheckpointId = checkpointId;
+ } else {
+ LOG.info(
+ "Skipping committing checkpoint {}. {} is already committed.",
+ checkpointId,
+ maxCommittedCheckpointId);
+ }
+ } finally {
+ // reload the table in case new configuration is needed
+ this.table = tableSupplier.get();
Review Comment:
Shall we update the `this.manifestOutputFileFactory.manifestOutputFileFactory` here, so we do not need to provide the table at every call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722270134
> Normally, I would think table load (dozens or hundreds of milliseconds?) is probably a magnitude or two more expensive than credentials refresh (a few milliseconds?). This is where I share the same concern that @pvary raised about every task manager refreshing/loading tables periodically.
>
> Can you elaborate a little how REST catalog makes table load efficient?
>
We've implemented several optimizations in our catalog so table load responses are around 24ms, including our API auth checks. We have refs-only mode on by default. We're continuing work to make that even more efficient.
> I also echo @pvary 's comment on the confusion btw `TableLoader` and `TableSupplier`. There are actually some questions on the design `TableLoader` (e.g. [not handling well close well](https://github.com/apache/iceberg/pull/6614)). I would be in favor a refactor/redesign of the table loader/catalog lifecycle in Flink. But that will probably be a big discussion by itself.
I agree here. It is possible to reuse the TableLoader interface with a couple of changes if that seems acceptable.
>
> I also thought you use pre-signed urls from REST catalog. with that, what is the credentials expiration problem here? S3 session credentials also have auto-refresh capability. Hence, I am trying to understand more about the credentials expiration and why is it tied to table object and needs refresh.
Refreshing credentials with the REST API can be done using OAuth2, which we use w/ the S3 signer, or using custom mechanism that passes credentials in the table config response, which are then passed to the FileIO during initialization. We use the latter mechanism for GCP tokens, Azure SAS tokens, S3 SigV4 exchange, and S3 vended credentials.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1333167322
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
> That said, I think if we agree that we are flexible about this part of the change, then we should not block this PR for this decision.
I was hoping to get some consensus on this topic/design (not implementation), because it affects the how we should expose this PR in the new public config `duration` vs `boolean. if we always want to tie the switch at checkpoint boundary, the later boolean version seems more natural because interval is not needed at all. this seems like the only open item left. internal implementation is easy to change. public API/config needs to be more careful.
@pvary do you see any fundamental problems if tasks make independent timing on switching to new schema (without checkpoint boundary alignment)? I am concerned about checkpoint boundary in the future implementation of coordinator broadcast. what if some broadcast msg came before the checkpoint barrier and some come after? is it really possible for us to fully align all writer tasks perfectly? I felt it is very challenging. the alternative approach is not to rely on it. let the write tasks switch independently. we just enhance the committer to handle potentially multiple partition specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1327011565
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java:
##########
@@ -41,15 +42,13 @@ class ManifestOutputFileFactory {
private final AtomicInteger fileCount = new AtomicInteger(0);
ManifestOutputFileFactory(
- TableOperations ops,
- FileIO io,
+ Supplier<Table> tableSupplier,
Review Comment:
Is this change needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722263748
> Implemenation-wise, a centralized solution for refreshing the table metadata is ideal. The implementation here, where the subtasks perform the refresh, still is works for many cases. If it is a point of contention, we can remove the implementation (ReloadingTableSupplier) but keep the ability to set one.
> Implementation-wise, a FileIO-specific API for credential refresh will still add load to an Iceberg REST implementation. Table load with the REST API and refs-only snapshot mode can be fairly efficient.
Normally, I would think table load (dozens or hundreds of milliseconds?) is probably a magnitude or two more expensive than credentials refresh (a few milliseconds?). This is where I share the same concern that @pvary raised about every task manager refreshing/loading tables periodically.
Can you elaborate a little how REST catalog makes table load efficient?
I also echo @pvary 's comment on the confusion btw `TableLoader` and `TableSupplier`. There are actually some questions on the design `TableLoader` (e.g. [not handling well close well](https://github.com/apache/iceberg/pull/6614)). I would be in favor a refactor/redesign of the table loader/catalog lifecycle in Flink. But that will probably be a big discussion by itself.
I also thought you use pre-signed urls from REST catalog. with that, what is the credentials expiration problem here? S3 session credentials also have auto-refresh capability. Hence, I am trying to understand more about the credentials expiration and why is it tied to table object and needs refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326407817
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -108,7 +130,10 @@ public RowDataTaskWriterFactory(
@Override
public void initialize(int taskId, int attemptId) {
this.outputFileFactory =
- OutputFileFactory.builderFor(table, taskId, attemptId).format(format).build();
+ OutputFileFactory.builderFor(initTable, taskId, attemptId)
+ .format(format)
+ .ioSupplier(() -> tableSupplier.get().io())
Review Comment:
This was updated, so the table reload is explicitly called now, rather than done as part of `get()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326412354
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -117,6 +145,8 @@ public TaskWriter<RowData> create() {
outputFileFactory,
"The outputFileFactory shouldn't be null if we have invoked the initialize().");
+ FileIO io = tableSupplier.get().io();
Review Comment:
Mainly so it could be reused in the different writer constructors. I went ahead and got rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326994677
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -581,16 +608,26 @@ static RowType toFlinkRowType(Schema schema, TableSchema requestedSchema) {
static IcebergStreamWriter<RowData> createStreamWriter(
Table table,
+ TableLoader tableLoader,
FlinkWriteConf flinkWriteConf,
RowType flinkRowType,
- List<Integer> equalityFieldIds) {
+ List<Integer> equalityFieldIds,
+ long minReloadIntervalMs) {
Review Comment:
My comment might be lost somewhere, but I think it would be better to receive the TableSupplier here, instead of creating a new "incidentally" same supplier than we do in the FlinkSink.
Also seeing the TableLoader/TableSupplier in one place mad me think how do we separate the functionality of the 2. Do we really need 2 objects providing/reloading tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1326414370
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java:
##########
@@ -149,6 +156,12 @@ public void initializeState(StateInitializationContext context) throws Exception
// Open the table loader and load the table.
this.tableLoader.open();
this.table = tableLoader.loadTable();
+ if (reloadIntervalMs > 0) {
+ this.tableSupplier = new ReloadingTableSupplier(table, tableLoader, reloadIntervalMs);
Review Comment:
`TableLoader` is specifically for loading a table from the source of truth, e.g. a catalog or HadoopTables. A `TableSupplier` implementation might get the table from someplace else, like the delegation framework or a broadcast (the initial TableSupplier implementation uses a TableLoader).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1716989137
I am very concerned about this simplified solution. We should consider:
- Concurrency issues, when the table metadata has been changed between reloads. Maybe we should only refresh the table metadata during checpoints, so we can localize, and handle changes in one place?
- Load issues, as every TaskManager with a writer task connects to the catalog, and does so periodically. This will become costly very soon. I would prioritize a centralized solution.
- The current default implementation does not solve the schema evolution problem, since that would mean on-demand table reload instead of periodic reload
If the main goal here is to refresh the credentials, then maybe a proxy solution above the accessor could be a better solution, like the https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java where we get a new Kerberos token when the previous one has expired.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324508651
##########
flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java:
##########
@@ -369,4 +369,25 @@ public void testOverrideWriteConfigWithUnknownFileFormat() {
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid file format: UNRECOGNIZED");
}
+
+ @Test
+ public void testWriteRowReloadEnabled() throws Exception {
+ List<Row> rows = Lists.newArrayList(Row.of(1, "hello"), Row.of(2, "world"), Row.of(3, "foo"));
+ DataStream<RowData> dataStream =
+ env.addSource(createBoundedSource(rows), ROW_TYPE_INFO)
+ .map(CONVERTER::toInternal, FlinkCompatibilityUtil.toTypeInfo(SimpleDataUtil.ROW_TYPE));
+
+ FlinkSink.forRowData(dataStream)
+ .table(table)
+ .tableLoader(tableLoader)
+ .writeParallelism(parallelism)
+ .reloadIntervalMs(1000)
Review Comment:
is this testing with a reload interval just to make sure nothing is failing? or would the behavior be different when table reloading would be disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1324794094
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -108,7 +130,10 @@ public RowDataTaskWriterFactory(
@Override
public void initialize(int taskId, int attemptId) {
this.outputFileFactory =
- OutputFileFactory.builderFor(table, taskId, attemptId).format(format).build();
+ OutputFileFactory.builderFor(initTable, taskId, attemptId)
+ .format(format)
+ .ioSupplier(() -> tableSupplier.get().io())
Review Comment:
Using the current table's FileIO to perform file operations should be safe, I think it actually desirable. For example, if write properties on the table are updated, those will be applied at runtime rather than requiring a restart (only if table reload is explicitly enabled).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #8555: Flink: provide table supplier in sink for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722139650
@bryanck I have 2 questions to understand the problem here
1. how is this handled in Spark? I assume credentials could be valid for a few hours and a Spark job can run for 10+ hours. so this problem also applies to Spark.
2. can the credentials refresh capability baked into `FileIO`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332388126
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
We may want to ensure the schema, partition spec, and Flink schema are in sync between the appender factory and writer when schema evolution is introduced? That isn't an issue for now as those use the initial table only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332405052
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
> We may want to ensure the schema, partition spec, and Flink schema are in sync between the appender factory and writer when schema evolution is introduced?
Once every writer task refreshed the table object for schema or partition spec change, the latest table object can be used directly by appender factory from the `Supplier<Table>`. do we need to align the cutover across all writer tasks at checkpoint boundary?
Why can't every writer task refresh and apply schema and partition spec change independently? I see the need for committer to handle multiple partition specs, which needs to be enhanced. what else?
> schema evolution
I see two forms of automatic schema evolution. Maybe a little diverted from the scope of this PR.
1. Iceberg table schema is automatically updated out-of-band (e.g. by control plane). but we want to avoid the need to redeploy the Flink job to pick up the latest schema in the Iceberg table. This is the simpler form and easier to support. In this form, typically Iceberg table schema is updated way before data started to flow with new schema.
2. Incoming data changed (e.g. a new column is added). Flink writer detected that the Iceberg table schema doesn't have the new column, update the Iceberg table schema, automatically start to write to the Iceberg table using the new schema. This is a bit more difficult to support as the parallel writers can compete to update Iceberg tables schema. In this scenario, there is no control plane or other automation to sync up data stream schema with Iceberg table schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332106904
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteOptions.java:
##########
@@ -64,4 +65,8 @@ private FlinkWriteOptions() {}
public static final ConfigOption<Integer> WRITE_PARALLELISM =
ConfigOptions.key("write-parallelism").intType().noDefaultValue();
+
+ @Experimental
+ public static final ConfigOption<Long> TABLE_REFRSH_MS =
+ ConfigOptions.key("table-refresh-ms").longType().noDefaultValue();
Review Comment:
I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1728134801
@stevenzwu We'd also need this to be set when using Flink SQL. I was considering removing `FlinkSink.tableRefreshInterval()` and using a table property instead, or perhaps a Flink configuration. Can you advise on what that best approach would be for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1332115658
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
We'll wait for others to reply, from my perspective, either one works for FileIO refresh, but for schema evolution I think you'd want it on checkpoint boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1730825750
@bryanck: Thanks for reaching out, and discussing the possible ways forward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1331928698
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.sink;
+
+import java.time.Duration;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+class CachingTableSupplier implements SerializableSupplier<Table> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CachingTableSupplier.class);
+
+ private final Table initialTable;
+ private final TableLoader tableLoader;
+ private final Duration tableRefreshInterval;
+ private long nextReloadTimeMs;
+ private transient Table table;
+
+ CachingTableSupplier(Table initialTable, TableLoader tableLoader, Duration tableRefreshInterval) {
+ Preconditions.checkArgument(initialTable != null, "initialTable cannot be null");
+ Preconditions.checkArgument(tableLoader != null, "tableLoader cannot be null");
+ Preconditions.checkArgument(
+ tableRefreshInterval != null, "tableRefreshInterval cannot be null");
+ this.initialTable = initialTable;
+ this.table = initialTable;
+ this.tableLoader = tableLoader;
+ this.tableRefreshInterval = tableRefreshInterval;
+ this.nextReloadTimeMs = System.currentTimeMillis() + tableRefreshInterval.toMillis();
+ }
+
+ @Override
+ public Table get() {
+ if (table == null) {
+ this.table = initialTable;
+ }
+ return table;
+ }
+
+ public void refresh() {
Review Comment:
Yes that is possible. The tasks won't necessarily be in sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks merged pull request #8555: Flink: table supplier in writer operator for optional table refresh
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks merged PR #8555:
URL: https://github.com/apache/iceberg/pull/8555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330740391
##########
core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java:
##########
@@ -158,12 +161,31 @@ public Builder suffix(String newSuffix) {
return this;
}
+ /**
+ * Configures a {@link FileIO} supplier, which is used to dynamically refresh the file IO
Review Comment:
nit: `which is used to` -> `which can potentially`
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/RowDataTaskWriterFactory.java:
##########
@@ -105,10 +104,17 @@ public RowDataTaskWriterFactory(
}
}
+ void setTable(Table table) {
Review Comment:
can we define a new constructor that takes in `Supplier<Table>`? the old constructor can simply forward to this new constructor. then `RefreshableTableSupplier` can be plugged in directly
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/TableLoader.java:
##########
@@ -38,6 +38,8 @@ public interface TableLoader extends Closeable, Serializable, Cloneable {
void open();
+ boolean isOpen();
Review Comment:
I guess we can revisit this when we moved back to supplier model.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/CachingTableLoader.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+@Experimental
+public class CachingTableLoader implements TableLoader {
Review Comment:
sorry. there are probably 2 reasons I changed my mind (1) I missed how `Supplier<Table> tableSupplier` was used like in `ManifestOutputFileFactory`. also (2) earlier, `TableSupplier` was exposed as public to users in `FlinkSink`. that can be confusing to users with `TableLoader` vs `TableSupplier`.
With it being non-public and hidden from users, I think `TableSupplier` makes more sense to me. We can pass the same object to `ManifestOutputFileFactory` and others directly. I just checked the commit history. I actually like the initial commit
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +269,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the interval for refreshing the table instance in {@link IcebergStreamWriter}. If not
+ * specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param refreshInterval the interval for refreshing the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder tableRefreshInternal(Duration refreshInterval) {
Review Comment:
typo: Internal -> Interval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bryanck commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1330782646
##########
core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java:
##########
@@ -158,12 +161,31 @@ public Builder suffix(String newSuffix) {
return this;
}
+ /**
+ * Configures a {@link FileIO} supplier, which is used to dynamically refresh the file IO
Review Comment:
Sure, thanks, I made this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329439775
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##########
@@ -266,6 +268,20 @@ public Builder upsert(boolean enabled) {
return this;
}
+ /**
+ * Sets the table loader used to refresh the table instance in {@link IcebergStreamWriter}. If
+ * not specified then the default behavior is to not refresh the table, and the initial table
+ * instance initialized is used for the lifetime of the job.
+ *
+ * @param newTableLoader the table loader to use to refresh the table in writer subtasks
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ @Experimental
+ public Builder writeTableLoader(TableLoader newTableLoader) {
Review Comment:
or this can be `tableRefreshInternal(Duration refreshInterval)`? if it is not, we go with the existing behavior, otherwise the `CachingTableLoader` is used to wrap the user provided loader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a diff in pull request #8555: Flink: table loader in writer operator for optional table reload
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #8555:
URL: https://github.com/apache/iceberg/pull/8555#discussion_r1329543555
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/CachingTableLoader.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A table loader that will only reload a table after a certain interval has passed. WARNING: This
+ * table loader should be used carefully when used with writer tasks. It could result in heavy load
+ * on a catalog for jobs with many writers.
+ */
+@Experimental
+public class CachingTableLoader implements TableLoader {
Review Comment:
after looking at other classes using `Supplier<Table>`, I think this could be called `RefreshableTableSupplier` that implements `Supplier<Table>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org