You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "vladislav-sidorovich (via GitHub)" <gi...@apache.org> on 2026/02/22 19:55:57 UTC
[PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
vladislav-sidorovich opened a new pull request, #15407:
URL: https://github.com/apache/iceberg/pull/15407
**Teste**:
In the following PRs, I will add all the tables from: [Delta golden tables](https://github.com/delta-io/delta/tree/master/connectors/golden-tables/src)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2844407313
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
Review Comment:
This can cause silent data corruption if the table has column mapping enabled. For context, Delta Lake supports three column mapping modes:
* `none`: Parquet files use the same column names as the logical schema.
* `name`: Parquet files use physical names (UUIDs or opaque strings) that differ from the logical names The Delta log stores a mapping between them. This enables column renames without rewriting
data files.
* `id`: Similar to name mode but maps by field ID instead of name.
Here we are mapping the schema based on the logical names. This won't work when column mapping is enabled (name or id). Delta Lake's UniForm feature requires column mapping to be enabled (name or id mode) and carries the physical-to-logical name mapping through to Iceberg via Iceberg's own column mapping (field IDs).
For now, we should at least validate that column mapping is not enabled and fail fast:
```
Map<String, String> configuration = deltaSnapshot.getMetadata().getConfiguration();
String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
if (!"none".equals(columnMappingMode)) {
throw new UnsupportedOperationException("...");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3144043530
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
Review Comment:
Correct, `RemoveFile` action contains DVs as well. I was thinking about this scenario in conversion and if I understand properly, DVs in remove action stay for state consistency to find a correct (P, DV) pair (or delete bin file as well). So it means that DVs in `RemoveFile` action could be ignored (Unless I miss something).
Could you share a table example or sequence of command how to generate input to reproduce the problem?
Btw. is this comment AI generated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143717249
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
Review Comment:
The intention is to support `VACUUM` operation as well and there is a dedicated test for it `org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#testConversionAfterVacuum`.
The logic is the very similar to the previous one:
1. Find first retractable commit => Create Iceberg commit.
2. Go 1 by 1 per Delta version after the recreatable commit.
The diff is in `commitDeltaSnapshotToIcebergTransaction` vs `convertEachDeltaVersion`. While methods are similar, the way how to collect required data files is different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143767808
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java:
##########
@@ -450,11 +448,12 @@ private void tagCurrentSnapshot(long deltaVersion, Transaction transaction) {
*/
private static String getFullFilePath(String path, String tableRoot) {
URI dataFileUri = URI.create(path);
- String decodedPath = URLDecoder.decode(path, StandardCharsets.UTF_8);
if (dataFileUri.isAbsolute()) {
- return decodedPath;
+ return dataFileUri.getScheme().equalsIgnoreCase("file") ? dataFileUri.getPath() : path;
} else {
- return tableRoot + File.separator + decodedPath;
+ String decodedPath = dataFileUri.getPath();
+ String separator = tableRoot.contains(":/") ? "/" : File.separator;
+ return tableRoot + (tableRoot.endsWith(separator) ? "" : separator) + decodedPath;
Review Comment:
It's the fix to use `File.separator` only for `real` file systems, but you are right. `BaseSnapshotDeltaLakeTableAction` should be completely replaced so I would prefer do not touch it.
Reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3451039645
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
Review Comment:
It's not really clear to me. Could you please explain a bit more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3567907428
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
Thanks for the explanation!
> In this test I intentionally delete some logs (not all) to make the delta versions not re-creatable and not continuous (started from table creation).
If in this test we did not delete the 3 version, seems the minimum "re-constructable" version will be `0` and during the conversion we will hit some `File NotFound` error due to VACUUM already cleaned up?
A similar scenario is that VACUUM cleaned a data file referenced in a version that not yet included in any checkpoint. E.g.
- Version 0: DataFile 0
- Version 1: DELETE DataFile 0, ADD DataFile 1
- VACUUM: deletes version 0's file
The earliest reconstructable version returned by kernel api would be `0` but it is not actually re-constructable due to missing data files.
This is also related to another comment https://github.com/apache/iceberg/pull/15407/changes#r3143717249 where we use the try-catch about find the actual starting version for conversion.
I understand that this is an edge case and not necessarily need to be addressed in this PR. Could we track it explicitly in the feature tracking issue/sub-issue. Please let me know WDYT : )
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
Review Comment:
+1, probably [ImmutableMapBuilder.buildKeepingLast()](https://guava.dev/releases/33.6.0-jre/api/docs/com/google/common/collect/ImmutableMap.Builder.html?utm_source=chatgpt.com#buildKeepingLast())
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
Review Comment:
+1 to at least fail early to avoid silent corruption. Could we include that in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3518744791
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
Review Comment:
I moved all the `internal` delta API usage to `InternalDeltaKernelUtils` so it will be easier to refactor it in the future.
Additionally, `InternalDeltaKernelUtils` now represents all the internal API that missed in public api and can be used as the list for contributions to `delta kernel` lib.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848538567
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
Review Comment:
Thank you for the input, it's very valuable. I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4111338416
@nastra since you kindly reviewed the earlier version, I'd love to get your thoughts on the updated core logic before I do the final refactoring to remove internal Delta classes.
@aokolnychyi Since you’ve contributed so much to the Deletion Vectors implementation in Iceberg, I wanted to reach out. Could you take a quick look at the DV conversion logic in my PR to make sure I’ve wired everything up correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848848338
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
Review Comment:
The internal APIs can change or disappear without any notice. I would think hard about avoiding dependencies on internal APIs, including changing semantics. (e.g. not preserving all the history by default).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-5837973231
@HonahX I hope all your inputs are resolved. Could you please take a look and approve the PR if you are ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4271277614
##########
dev/.rat-excludes:
##########
@@ -29,4 +29,5 @@ sitemap.xml
.python-version
**/*_index.md
**/.venv/**
+**/_delta_log/_last_checkpoint
Review Comment:
There is https://github.com/delta-incubator/dat. We can download and cache it locally in `~./m2/..` for the future builds.
I think it's the way to go:
1. No source in Iceberg repo (deps as artifact)
2. Static immutable files for Delta protocol
3. Downloaded once and cached as regular dependency.
@kevinjqliu are you ok with the plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3144076214
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
Review Comment:
> Could you share a table example or sequence of command how to generate input to reproduce the problem?
Here is an gist with example: https://gist.github.com/laskoviymishka/065a6754edda3db86c8caf96216e3128
> Btw. is this comment AI generated?
I used Claude to navigate this PR and iterate through the review pass; the conclusions and reproducer below are mine after stepping through the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2880659210
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeActionsTranslationUtil.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.data.Row;
+import io.delta.kernel.internal.actions.AddFile;
+
+/**
+ * Util class helps to handle json operations for <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">delta action</a>
+ */
+public class DeltaLakeActionsTranslationUtil {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3572883955
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
Review Comment:
Plz don't use Table API but TableManager
https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/TableManager.java
The previous one is kind of deprecated as it is not supported catalog managed table for Delta https://delta.io/blog/2026-02-02-delta-catalog-managed-tables
Delta is also moving to catalog oriented. so using Table API will end up Kernel unable to recognize catalog managed commits(staged comment that is not yet backfilled) causing ether fail query or data lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3569550170
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
> If in this test we did not delete the 3 version, seems the minimum "re-constructable" version will be 0 and during the conversion we will hit some File NotFound error due to VACUUM already cleaned up?
Correct, it will be `File NotFound errors`. These errors still present because I intentionally do not delete all `non-reconstructable` versions. The code will find version `4` it will be an error and move on till first `reconstractable` which is`10` and asserted bellow in `Assert` section.
So, this test clarify more or less real-world scenario.
> This is also related to another comment https://github.com/apache/iceberg/pull/15407/changes#r3143717249
I believe this edge case is covered, but let me double check it and come back here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3518864913
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
Review Comment:
refactored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3518880090
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
Review Comment:
refactord
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
Review Comment:
refactored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6279456888
Thanks both, lets figure out the next steps for this PR here and move the future convo to devlist.
I'll be happy to merge this once we get rid of the 100+ test files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4372208812
##########
dev/.rat-excludes:
##########
@@ -29,4 +29,5 @@ sitemap.xml
.python-version
**/*_index.md
**/.venv/**
+**/_delta_log/_last_checkpoint
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3144043530
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
Review Comment:
Correct, `RemoveFile` action contains DVs as well. I was thinking about this scenario in conversion and if I understand properly, at most one active DV can exist per data file and DVs in remove action stay for state consistency to find a correct (P, DV) pair (or delete bin file as well). So it means that DVs in `RemoveFile` action could be ignored (Unless I miss something).
Could you share a table example or sequence of command how to generate input to reproduce the problem?
Btw. is this comment AI generated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143854239
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
Review Comment:
There is `unit-test` to cover all possible data types `TestDeltaLakeKernelTypeToType` including structs.
Unpartitioned table also covered by `unit-test` in `TestBaseSnapshotDeltaLakeKernelTableAction`. Actually, I tried to move more logic to unit tests.
I will also add more tables from https://github.com/delta-io/delta/tree/master/connectors/golden-tables/src , so `unit-test` will be robust. I removed these tables from the initial PR to reduce number of files in the PR.
Regarding `checkDataFilePathsIntegrity` I'm not 100% sure. There is no simply utility method that will help to collect all data files, so it means we will need to introduce a `logic` into the test similar to one we have in the conversion. At the same time this test has `checkSnapshotIntegrityForQuery` which verify full data for the table per tag/version. Could you clarify what test scenarios is missing and what do you want to test more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143807423
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143786306
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeletionVectorConverter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.internal.actions.DeletionVectorDescriptor;
+import io.delta.kernel.internal.deletionvectors.DeletionVectorUtils;
+import io.delta.kernel.internal.deletionvectors.RoaringBitmapArray;
+import io.delta.kernel.internal.util.Tuple2;
+
+public class DeletionVectorConverter {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3136208821
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
Review Comment:
Setting the validate-from snapshot to currentSnapshot() disables RowDelta's conflict validation wholesale, which is what's masking the Add+Remove-same-path bug above in tests. Please remove this or scope it to the narrower concern (e.g. validateDataFilesExist on a specific set) and let the real validation run.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
Review Comment:
Per Delta PROTOCOL.md, a DV update emits Add(P, dv=new) + Remove(P, dv=old) for the same path in the same commit. Here that lands as rowDelta.addRows(P) + removeRows(P) + addDeletes(puffin_for_P). This either fails validation or produces a snapshot where the file is added and removed simultaneously and the new DV dangles.
Additionally, prior Puffin DVs written for earlier versions of P are never retired — there is no removeDeletes(...). Please detect same-path Add+Remove pairs and emit a DV-only update (addDeletes + removeDeletes) without addRows/removeRows. This is the central DELETE/MERGE/UPDATE path on DV-enabled Delta tables and is currently incorrect.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
Review Comment:
Each ColumnarBatch triggers its own newAppend/newOverwrite/newRowDelta.commit().
A single Delta version can span multiple batches; an Add in batch 1 and its matching Remove in batch 2 produce two separate Iceberg snapshots with different sequence numbers (so DVs attach to the wrong add), and only the LAST snapshot gets the delta-version-N tag — time-travel by tag will see partial state. Please buffer all actions for a version across batches and emit one Iceberg commit per Delta version.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
Review Comment:
Only commitInfo, add, and remove are handled. metaData, protocol, cdc, txn are silently skipped even though DeltaLakeActionsTranslationUtil exposes isMetaData/isProtocol/isCdc/isTxn helpers (dead code today).
A mid-history metaData (ADD COLUMN, type widen) leaves Iceberg pinned to the earliest-recreatable schema while later AddFiles reference data written under a newer parquet schema → mis-mapped columns and wrong metrics at read time.
At minimum, fail fast when these appear at version > minimalAvailableDeltaVersion.
##########
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.ContentFileUtil;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+public class TestBaseSnapshotDeltaLakeKernelTableAction {
+ private static final String DELTA_GOLDEN_TABLES_ROOT = "delta/golden/";
+
+ @TempDir private File sourceDeltaFolder;
+ @TempDir private File icebergWarehouseFolder;
+ private String sourceTableLocation;
+ private final Configuration testHadoopConf = new Configuration();
+ private String newTableLocation;
+ private Catalog testCatalog;
+
+ @BeforeEach
+ public void before() throws IOException {
+ sourceTableLocation = sourceDeltaFolder.toURI().toString();
+
+ Path icebergTablePath = icebergWarehouseFolder.toPath().resolve("iceberg_table");
+ newTableLocation = icebergTablePath.toString();
+ Files.createDirectories(icebergTablePath);
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, icebergWarehouseFolder.toString());
+
+ HadoopCatalog icebergCatalog = new HadoopCatalog();
+ icebergCatalog.setConf(new Configuration());
+ icebergCatalog.initialize("hadoop_catalog", properties);
+ testCatalog = icebergCatalog;
+ }
+
+ @Test
+ public void testRequiredTableIdentifier() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .icebergCatalog(testCatalog)
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredIcebergCatalog() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredDeltaLakeConfiguration() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Make sure to configure the action with a valid deltaLakeConfiguration");
+ }
+
+ @Test
+ public void testDeltaTableColumnMappingFeatureDisabled() throws Exception {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Transaction txn =
+ createEmptyDeltaTableTransaction(engine)
+ .withTableProperties(engine, Map.of("delta.columnMapping.mode", "name"))
+ .build(engine);
+ txn.commit(engine, CloseableIterable.emptyIterable());
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act & check
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Conversion of Delta Lake tables with columnMapping feature is not supported.");
+ }
+
+ @Test
+ public void testEmptyTableConversion() {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ createEmptyDeltaTableTransaction(engine)
+ .build(engine)
+ .commit(engine, CloseableIterable.emptyIterable());
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+
+ org.apache.iceberg.Table table = testCatalog.loadTable(icebergTable);
+ assertThat(((BaseTable) table).operations().current().formatVersion()).isEqualTo(3);
+ }
+
+ @Test
+ public void testTableCreated() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+ }
+
+ @Test
+ public void testDefaultTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("original_location")).isEqualTo(sourceTableLocation);
+ assertThat(properties.get("snapshot_source")).isEqualTo("delta");
+ assertThat(properties.containsKey(TableProperties.DEFAULT_NAME_MAPPING)).isTrue();
+ }
+
+ @Test
+ public void testCustomTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation)
+ .tableProperty("custom_prop_1", "custom val 1")
+ .tableProperty("custom_prop_2", "custom val 2")
+ .tableProperties(
+ ImmutableMap.of(
+ "custom_map_prop1", "val 1",
+ "custom_map_prop2", "val 2"));
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("custom_prop_1")).isEqualTo("custom val 1");
+ assertThat(properties.get("custom_prop_2")).isEqualTo("custom val 2");
+ assertThat(properties.get("custom_map_prop1")).isEqualTo("val 1");
+ assertThat(properties.get("custom_map_prop2")).isEqualTo("val 2");
+ }
+
+ @Test
+ void testDeltaTableNotExist() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Delta Lake table does not exist at the given location: %s", sourceTableLocation);
+ }
+
+ @Test
+ public void testDeltaTableDVSupported() throws Exception {
Review Comment:
This only checks hasDeleteFiles == true and isDV(deleteFile). It never scans the converted Iceberg table to confirm row counts match the post-DV Delta row count, or that the specific rows Delta marked deleted are absent from Iceberg reads.
The dv-partitioned-with-checkpoint golden exercises DV updates and file removes (V6–V13) — please assert row-level parity (count + sampled content) so the bugs in comments 1–3 cannot pass silently.
Same concern for goldenDeltaTableConversion at line 322–335, which only asserts execute() doesn't throw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3493465074
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
Review Comment:
I'm not sure I get an issue with the files colliding paths. This method used for each DV puffin file generation: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java#L106
So, `fileCount.incrementAndGet()` will always generate next sequential (and unique) file name. It seems that `partitionId=1, taskId=1` are need for execution in cluster (on different workers), with the current conversion scenario the usage seems valid. Do I miss anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2855609897
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
Review Comment:
Yes, it was kind of debug log. Removed.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
Review Comment:
fixed, tnx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4679543288
@vladislav-sidorovich is this ready to one more round of review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6097770543
Hey guys, @yyanyy , @jackye1995 , @nastra , @danielcweeks , @RussellSpitzer
you reviewed the previous version of this module. Could you please take a look on this update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848478386
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
Review Comment:
No, there are no public API available for these purposes we need.
Yes, I want go through table history step by step, so we will have exactly the same granularity in the history.
At the same time it's quite safe to use an internal API because it's depends on the Delta protocol which is stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848511442
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
Review Comment:
It's necessary because I use internal API below. [getChanges](https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableImpl.java#L184 ) API is available only in `TableImpl` but not in the `Table` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3518908697
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3518930283
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeletionVectorConverter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.internal.actions.DeletionVectorDescriptor;
+import io.delta.kernel.internal.deletionvectors.DeletionVectorUtils;
+import io.delta.kernel.internal.deletionvectors.RoaringBitmapArray;
+import io.delta.kernel.internal.util.Tuple2;
Review Comment:
It's return type of `io.delta.kernel.internal.deletionvectors.DeletionVectorUtils#loadNewDvAndBitmap`, so I can't really avoid to import it.
All delta internal api usage refactored and moved to a separate class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3491111844
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
+ } else if (!dataFilesToAdd.isEmpty() && dataFilesToRemove.isEmpty()) {
+ // Append only
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ } else if (dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Delete only
+ DeleteFiles deleteFiles = transaction.newDelete();
+ dataFilesToRemove.forEach(deleteFiles::deleteFile);
+ deleteFiles.commit();
+ } else if (!dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Overwrite
+ OverwriteFiles overwriteFiles = transaction.newOverwrite();
+ dataFilesToAdd.forEach(overwriteFiles::addFile);
+ dataFilesToRemove.forEach(overwriteFiles::deleteFile);
+ overwriteFiles.commit();
+ } else {
+ // Tag case
+ transaction.newAppend().commit();
+ }
+ }
+
+ private DataFile buildDataFileFromAddDeltaAction(AddFile addFile, Transaction transaction) {
+ String path = addFile.getPath();
+ long dataFileSize = addFile.getSize();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
+ throw new NotFoundException(
+ "File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage",
+ fullFilePath, deltaTableLocation);
+ }
+
+ MetricsConfig metricsConfig = MetricsConfig.forTable(transaction.table());
+ String nameMappingString =
+ transaction.table().properties().get(TableProperties.DEFAULT_NAME_MAPPING);
+ NameMapping nameMapping =
+ nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ // TODO read metrics from Delta log to avoid data flies read
+ Metrics metrics = ParquetUtil.fileMetrics(inputDataFile, metricsConfig, nameMapping);
+
+ Map<String, String> partitionValues = VectorUtils.toJavaMap(addFile.getPartitionValues());
+ PartitionSpec partitionSpec = transaction.table().spec();
+ List<String> partitionValueList =
+ partitionSpec.fields().stream()
+ .map(PartitionField::name)
+ .map(partitionValues::get)
+ .collect(Collectors.toList());
+
+ return DataFiles.builder(partitionSpec)
+ .withPath(fullFilePath)
+ .withFormat(FileFormat.PARQUET) // Delta supports only parquet datafiles
+ .withFileSizeInBytes(dataFileSize)
+ .withMetrics(metrics)
+ .withPartitionValues(partitionValueList)
+ .build();
+ }
+
+ private DataFile buildDataFileFromRemoveDeltaAction(
+ RemoveFile removeFile, Transaction transaction) {
+ String path = removeFile.getPath();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
Review Comment:
I think there is a misunderstanding here.
1. `VACUUM` is fully supported and covered by tests.
2. This check verifies that data file exists not because it was deleted by `VACUUM` but in general verify integrity of the current table snapshot. Also, this data file used below in `ParquetUtil.fileMetrics`, so it's prevention of unclear not-found error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2955037892
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
Review Comment:
Checking the latest snapshot is sufficient here. Column mapping can technically be dropped (via `DROP FEATURE` in Delta), but the drop process rewrites all data files to use logical column names and the old files are removed. So if the latest snapshot has mode = none, the data files accessible through available snapshots will use logical names. No need to iterate through all snapshots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4676152768
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3450928683
##########
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.ContentFileUtil;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+public class TestBaseSnapshotDeltaLakeKernelTableAction {
+ private static final String DELTA_GOLDEN_TABLES_ROOT = "delta/golden/";
+
+ @TempDir private File sourceDeltaFolder;
+ @TempDir private File icebergWarehouseFolder;
+ private String sourceTableLocation;
+ private final Configuration testHadoopConf = new Configuration();
+ private String newTableLocation;
+ private Catalog testCatalog;
+
+ @BeforeEach
+ public void before() throws IOException {
+ sourceTableLocation = sourceDeltaFolder.toURI().toString();
+
+ Path icebergTablePath = icebergWarehouseFolder.toPath().resolve("iceberg_table");
+ newTableLocation = icebergTablePath.toString();
+ Files.createDirectories(icebergTablePath);
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, icebergWarehouseFolder.toString());
+
+ HadoopCatalog icebergCatalog = new HadoopCatalog();
+ icebergCatalog.setConf(new Configuration());
+ icebergCatalog.initialize("hadoop_catalog", properties);
+ testCatalog = icebergCatalog;
+ }
+
+ @Test
+ public void testRequiredTableIdentifier() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .icebergCatalog(testCatalog)
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredIcebergCatalog() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredDeltaLakeConfiguration() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Make sure to configure the action with a valid deltaLakeConfiguration");
+ }
+
+ @Test
+ public void testDeltaTableColumnMappingFeatureDisabled() throws Exception {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Transaction txn =
+ createEmptyDeltaTableTransaction(engine)
+ .withTableProperties(engine, Map.of("delta.columnMapping.mode", "name"))
+ .build(engine);
+ txn.commit(engine, CloseableIterable.emptyIterable());
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act & check
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Conversion of Delta Lake tables with columnMapping feature is not supported.");
+ }
+
+ @Test
+ public void testEmptyTableConversion() {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ createEmptyDeltaTableTransaction(engine)
+ .build(engine)
+ .commit(engine, CloseableIterable.emptyIterable());
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+
+ org.apache.iceberg.Table table = testCatalog.loadTable(icebergTable);
+ assertThat(((BaseTable) table).operations().current().formatVersion()).isEqualTo(3);
+ }
+
+ @Test
+ public void testTableCreated() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+ }
+
+ @Test
+ public void testDefaultTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("original_location")).isEqualTo(sourceTableLocation);
+ assertThat(properties.get("snapshot_source")).isEqualTo("delta");
+ assertThat(properties.containsKey(TableProperties.DEFAULT_NAME_MAPPING)).isTrue();
+ }
+
+ @Test
+ public void testCustomTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation)
+ .tableProperty("custom_prop_1", "custom val 1")
+ .tableProperty("custom_prop_2", "custom val 2")
+ .tableProperties(
+ ImmutableMap.of(
+ "custom_map_prop1", "val 1",
+ "custom_map_prop2", "val 2"));
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("custom_prop_1")).isEqualTo("custom val 1");
+ assertThat(properties.get("custom_prop_2")).isEqualTo("custom val 2");
+ assertThat(properties.get("custom_map_prop1")).isEqualTo("val 1");
+ assertThat(properties.get("custom_map_prop2")).isEqualTo("val 2");
+ }
+
+ @Test
+ void testDeltaTableNotExist() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Delta Lake table does not exist at the given location: %s", sourceTableLocation);
+ }
+
+ @Test
+ public void testDeltaTableDVSupported() throws Exception {
Review Comment:
The integration test has this assert org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#checkTagContentAndOrder and this assert verify number of tag/versions in delta and Iceberg and it also verify the data by `SELECT * AS OF VERSION`.
So, there is no bug for DVs related the the same data file (as we discussed offline) and there is test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3830655258
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
Fixed.
Test `org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#testConversionAfterVacuumNoCheckpoints` added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6403773915
> Thanks both, lets figure out the next steps for this PR here and move the future convo to devlist.
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848490000
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
Review Comment:
Not really. I do not expect this code to be merged/used before I will add handling of all other Delta Actions.
So, I will support of `remove` action quite soon and fail fast will not be needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143827721
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
The short answers:
1. No, logs clean-up is not the same as VACUUM
2. The conversation will **not** failed.
Explanations:
The logs clean-up and VACUUM are 2 parallel and different processes. In this test I intentionally delete some logs (not all) to make the delta versions `not re-creatable` and not continuous (started from table creation). For example in this test Delta versions `3-9` are exist but are not re-creatable because the data files were deleted by VACUUM.
So the first re-creatable version in this test is `10`, this version used as initial version in the conversion and the test is for that scenario.
To sum-up:
1. `VACUUM` and `logs clean-up` are 2 different processes.
2. In this test I first execute `VACUUM` with `RETAIN 0 HOURS`.
3. After some DML operation I simulated (as written in the comment) logs `clean-up`.
4. Verify that conversion works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143947761
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
Review Comment:
Yes, it's intentionally as written in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143940549
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
Review Comment:
I agree, fixed. Now it's 1 delta version to 1 iceberg commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3572954949
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.delta.InternalDeltaKernelUtils.DeltaAddFile;
+import org.apache.iceberg.delta.InternalDeltaKernelUtils.DeltaRemoveFile;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String CONVERSION_TOOL_PROP = "conversion_tool";
+ private static final String CONVERSION_TOOL_VALUE = "iceberg-delta-lake";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private Table deltaTable;
+ private Snapshot deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ Snapshot initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return ImmutableSnapshotDeltaLakeTable.Result.builder()
+ .snapshotDataFilesCount(processedDataFiles.size())
+ .build();
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ Snapshot deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ InternalDeltaKernelUtils.changes(
+ deltaTable, deltaEngine, currDeltaVersion, currDeltaVersion)) {
Review Comment:
Similarly https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/TableManager.java#L76-L78 will be the replacement of `Table.getChanges`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3467936288
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
Review Comment:
Table.forPath is actually an old api in delta kernel so few efforts to maintenance or improve will be on that end.
The new one is more snapshot centric - TableManager.loadSnapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3492255871
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
Review Comment:
Good point, let me rewrite it to rely on `metaData`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3569865980
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
Review Comment:
I believe the current implementation is a better option because it's fail fast pattern.
1. `execute()` must be called only once, the second call of execute is not expected.
1. multiple properties can be added only by mistake and an exception will be thrown before the conversion will start. I added new unit-test `testCustomTablePropertiesOverlap` to explicitly cover this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4166832723
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.delta.InternalDeltaKernelUtils.DeltaAddFile;
+import org.apache.iceberg.delta.InternalDeltaKernelUtils.DeltaRemoveFile;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String CONVERSION_TOOL_PROP = "conversion_tool";
+ private static final String CONVERSION_TOOL_VALUE = "iceberg-delta-lake";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private Table deltaTable;
+ private Snapshot deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ Snapshot initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return ImmutableSnapshotDeltaLakeTable.Result.builder()
+ .snapshotDataFilesCount(processedDataFiles.size())
+ .build();
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ Snapshot deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ InternalDeltaKernelUtils.changes(
+ deltaTable, deltaEngine, currDeltaVersion, currDeltaVersion)) {
Review Comment:
Thanks. Added as item to the plan https://github.com/apache/iceberg/issues/17146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2880656969
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
Review Comment:
Support of `Remove` action was added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3144187091
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
Review Comment:
Yes, but my ask here: add a fail-fast on isMetaData/isProtocol past minimalAvailableDeltaVersion so users hit a clear error instead of a silently mis-converted table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3144201031
##########
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.ContentFileUtil;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+public class TestBaseSnapshotDeltaLakeKernelTableAction {
+ private static final String DELTA_GOLDEN_TABLES_ROOT = "delta/golden/";
+
+ @TempDir private File sourceDeltaFolder;
+ @TempDir private File icebergWarehouseFolder;
+ private String sourceTableLocation;
+ private final Configuration testHadoopConf = new Configuration();
+ private String newTableLocation;
+ private Catalog testCatalog;
+
+ @BeforeEach
+ public void before() throws IOException {
+ sourceTableLocation = sourceDeltaFolder.toURI().toString();
+
+ Path icebergTablePath = icebergWarehouseFolder.toPath().resolve("iceberg_table");
+ newTableLocation = icebergTablePath.toString();
+ Files.createDirectories(icebergTablePath);
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, icebergWarehouseFolder.toString());
+
+ HadoopCatalog icebergCatalog = new HadoopCatalog();
+ icebergCatalog.setConf(new Configuration());
+ icebergCatalog.initialize("hadoop_catalog", properties);
+ testCatalog = icebergCatalog;
+ }
+
+ @Test
+ public void testRequiredTableIdentifier() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .icebergCatalog(testCatalog)
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredIcebergCatalog() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredDeltaLakeConfiguration() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Make sure to configure the action with a valid deltaLakeConfiguration");
+ }
+
+ @Test
+ public void testDeltaTableColumnMappingFeatureDisabled() throws Exception {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Transaction txn =
+ createEmptyDeltaTableTransaction(engine)
+ .withTableProperties(engine, Map.of("delta.columnMapping.mode", "name"))
+ .build(engine);
+ txn.commit(engine, CloseableIterable.emptyIterable());
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act & check
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Conversion of Delta Lake tables with columnMapping feature is not supported.");
+ }
+
+ @Test
+ public void testEmptyTableConversion() {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ createEmptyDeltaTableTransaction(engine)
+ .build(engine)
+ .commit(engine, CloseableIterable.emptyIterable());
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+
+ org.apache.iceberg.Table table = testCatalog.loadTable(icebergTable);
+ assertThat(((BaseTable) table).operations().current().formatVersion()).isEqualTo(3);
+ }
+
+ @Test
+ public void testTableCreated() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+ }
+
+ @Test
+ public void testDefaultTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("original_location")).isEqualTo(sourceTableLocation);
+ assertThat(properties.get("snapshot_source")).isEqualTo("delta");
+ assertThat(properties.containsKey(TableProperties.DEFAULT_NAME_MAPPING)).isTrue();
+ }
+
+ @Test
+ public void testCustomTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation)
+ .tableProperty("custom_prop_1", "custom val 1")
+ .tableProperty("custom_prop_2", "custom val 2")
+ .tableProperties(
+ ImmutableMap.of(
+ "custom_map_prop1", "val 1",
+ "custom_map_prop2", "val 2"));
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("custom_prop_1")).isEqualTo("custom val 1");
+ assertThat(properties.get("custom_prop_2")).isEqualTo("custom val 2");
+ assertThat(properties.get("custom_map_prop1")).isEqualTo("val 1");
+ assertThat(properties.get("custom_map_prop2")).isEqualTo("val 2");
+ }
+
+ @Test
+ void testDeltaTableNotExist() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Delta Lake table does not exist at the given location: %s", sourceTableLocation);
+ }
+
+ @Test
+ public void testDeltaTableDVSupported() throws Exception {
Review Comment:
Yes, testConversionWithDeletionVectors does compare row contents, but It only exercises a single UPDATE (one DV-update commit).
The DV-update bugs surface on:
1. Two consecutive DV-updates on the same file:
- first UPDATE writes DV1
- second UPDATE writes DV2 with no retraction of DV1 → orphan Puffin, violates the v3 "≤1 DV per data file" invariant. SELECT * on the final snapshot can still pass while the metadata is broken.
2. Time-travel via delta-version-N tags: the test asserts tag presence but not that SELECT * AS OF VERSION delta-version-K matches Delta's VERSION AS OF K. Per-version correctness is the actual user-facing contract for this conversion.
Maybe worth to extend the integration test with those, both are a few lines on top of the existing fixture.
I'm not sure that this can be opt-out here, since it may highlight a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2844052662
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/DeltaLakeToIcebergMigrationSparkIntegration.java:
##########
@@ -60,4 +60,35 @@ static SnapshotDeltaLakeTable snapshotDeltaLakeTable(
.deltaLakeConfiguration(spark.sessionState().newHadoopConf())
.icebergCatalog(Spark3Util.loadIcebergCatalog(spark, catalogAndIdent.catalog().name()));
}
+
+ /**
+ * Example of how to use a {@link SparkSession}, a table identifier and a delta table location to
+ * construct an action for snapshotting the delta table to an iceberg table.
+ *
+ * @param spark a SparkSession with iceberg catalog configured.
+ * @param newTableIdentifier can be both 2 parts and 3 parts identifier, if it is 2 parts, the
+ * default spark catalog will be used
+ * @param deltaTableLocation the location of the delta table
+ * @return an instance of snapshot delta lake table action.
+ */
+ static SnapshotDeltaLakeTable snapshotDeltaLakeKernelTable(
+ SparkSession spark, String newTableIdentifier, String deltaTableLocation) {
+ Preconditions.checkArgument(
+ spark != null, "The SparkSession cannot be null, please provide a valid SparkSession");
+ Preconditions.checkArgument(
+ newTableIdentifier != null,
+ "The table identifier cannot be null, please provide a valid table identifier for the new iceberg table");
+ Preconditions.checkArgument(
+ deltaTableLocation != null,
+ "The delta lake table location cannot be null, please provide a valid location of the delta lake table to be snapshot");
Review Comment:
Nit: Replace with Delta Lake and Iceberg in the error messages.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
Review Comment:
Replace with `throw new UncheckedIOException(e);` (I think there is a CI failure related to this)
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
Review Comment:
Did you mean to remove this line?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
Review Comment:
We are using internal APIs of the kernel. This is fragile - can we refactor this to use the public APIs instead? `Snapshot`, `Table` etc. Or are we doing this because we are trying to preserve the table history during the conversion? I would try to avoid this as much as possible.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+ } else if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ commitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ }
+ }
+ }
+
+ // TODO Append only now
+ if (!dataFilesToAdd.isEmpty()) {
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ }
+
+ tagCurrentSnapshot(deltaVersion, commitTimestamp, transaction);
+
+ return dataFilesToAdd.size();
+ }
+
+ // TODO support more actions
+ private DataFile buildDataFileFromDeltaAction(AddFile addFile, Transaction transaction) {
+ String path = addFile.getPath();
+ long dataFileSize = addFile.getSize();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
+ throw new NotFoundException(
+ "File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage",
+ fullFilePath, deltaTableLocation);
+ }
+
+ MetricsConfig metricsConfig = MetricsConfig.forTable(transaction.table());
+ String nameMappingString =
+ transaction.table().properties().get(TableProperties.DEFAULT_NAME_MAPPING);
+ NameMapping nameMapping =
+ nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ Metrics metrics = ParquetUtil.fileMetrics(inputDataFile, metricsConfig, nameMapping);
+
+ Map<String, String> partitionValues = VectorUtils.toJavaMap(addFile.getPartitionValues());
+ PartitionSpec partitionSpec = transaction.table().spec();
+ List<String> partitionValueList =
+ partitionSpec.fields().stream()
+ .map(PartitionField::name)
+ .map(partitionValues::get)
+ .collect(Collectors.toList());
+
+ return DataFiles.builder(partitionSpec)
+ .withPath(fullFilePath)
+ .withFormat(FileFormat.PARQUET) // Delta supports only parquet datafiles
+ .withFileSizeInBytes(dataFileSize)
+ .withMetrics(metrics)
+ .withPartitionValues(partitionValueList)
+ .build();
+ }
+
+ @Nonnull
+ private static Schema convertToIcebergSchema(StructType deltaSchema) {
+ Type converted = new DeltaLakeKernelTypeToType(deltaSchema).convertType();
+ return new Schema(converted.asNestedType().asStructType().fields());
+ }
+
+ private long getEarliestDeltaLog() {
+ try {
+ // "_delta_log" is unmodifiable logs location
+ return DeltaHistoryManager.getEarliestDeltaFile(
+ deltaEngine, new Path(deltaTableLocation, "_delta_log"));
+ } catch (TableNotFoundException e) {
+ throw deltaTableNotFoundException(e);
+ }
+ }
+
+ private SnapshotImpl getLatestDeltaSnapshot() {
+ Snapshot latestSnapshot;
+ try {
+ latestSnapshot = deltaTable.getLatestSnapshot(deltaEngine);
+
+ assertSnapshotImpl(latestSnapshot);
+
+ return (SnapshotImpl) latestSnapshot;
+ } catch (TableNotFoundException e) {
+ throw deltaTableNotFoundException(e);
+ }
+ }
+
+ private PartitionSpec buildPartitionSpec(Schema schema, List<String> partitionNames) {
+ if (partitionNames.isEmpty()) {
+ return PartitionSpec.unpartitioned();
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(schema);
+ for (String partitionName : partitionNames) {
+ builder.identity(partitionName);
+ }
+ return builder.build();
+ }
+
+ private Map<String, String> buildTablePropertiesWithDelta(
+ SnapshotImpl deltaSnapshot, String originalLocation) {
+ icebergPropertiesBuilder.putAll(
+ org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.of(
+ SNAPSHOT_SOURCE_PROP, DELTA_SOURCE_VALUE, ORIGINAL_LOCATION_PROP, originalLocation));
+
+ Map<String, String> configuration = deltaSnapshot.getMetadata().getConfiguration();
+ icebergPropertiesBuilder.putAll(configuration);
+
+ return icebergPropertiesBuilder.build();
+ }
+
+ private void tagCurrentSnapshot(
+ long deltaVersion, Long deltaVersionTimestamp, Transaction transaction) {
+ long currentSnapshotId = transaction.table().currentSnapshot().snapshotId();
+
+ ManageSnapshots manageSnapshots = transaction.manageSnapshots();
+ manageSnapshots.createTag(DELTA_VERSION_TAG_PREFIX + deltaVersion, currentSnapshotId);
+
+ if (deltaVersionTimestamp != null) {
+ manageSnapshots.createTag(
+ DELTA_TIMESTAMP_TAG_PREFIX + deltaVersionTimestamp, currentSnapshotId);
+ }
+ manageSnapshots.commit();
+ }
+
+ private static void assertSnapshotImpl(Snapshot latestSnapshot) {
+ if (!(latestSnapshot instanceof SnapshotImpl)) {
+ throw new IllegalStateException(
+ "Unsupported impl of delta Snapshot: " + latestSnapshot.getClass());
+ }
+ }
+
+ @Nonnull
+ private IllegalArgumentException deltaTableNotFoundException(TableNotFoundException exception) {
+ return new IllegalArgumentException(
+ String.format(
+ "Delta Lake table does not exist at the given location: %s", deltaTableLocation),
+ exception);
+ }
+
+ private static String getFullFilePath(String path, String tableRoot) {
+ URI dataFileUri = URI.create(path);
+ String decodedPath = URLDecoder.decode(path, StandardCharsets.UTF_8);
+ if (dataFileUri.isAbsolute()) {
+ return decodedPath;
+ } else {
+ return tableRoot + File.separator + decodedPath;
Review Comment:
File.separator is `\` on Windows, which will produce invalid paths for cloud storage. This problem is there in the existing code, but let's fix it.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
Review Comment:
Typo in `delta log`
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeKernelTypeToType.java:
##########
@@ -16,47 +16,74 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.iceberg.spark;
-
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.types.ArrayType;
+import io.delta.kernel.types.BinaryType;
+import io.delta.kernel.types.BooleanType;
+import io.delta.kernel.types.ByteType;
+import io.delta.kernel.types.DataType;
+import io.delta.kernel.types.DateType;
+import io.delta.kernel.types.DecimalType;
+import io.delta.kernel.types.DoubleType;
+import io.delta.kernel.types.FieldMetadata;
+import io.delta.kernel.types.FloatType;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.LongType;
+import io.delta.kernel.types.MapType;
+import io.delta.kernel.types.ShortType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructField;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.types.TimestampNTZType;
+import io.delta.kernel.types.TimestampType;
+import io.delta.kernel.types.VariantType;
import java.util.List;
+import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
-import org.apache.spark.sql.types.ArrayType;
-import org.apache.spark.sql.types.BinaryType;
-import org.apache.spark.sql.types.BooleanType;
-import org.apache.spark.sql.types.ByteType;
-import org.apache.spark.sql.types.CharType;
-import org.apache.spark.sql.types.DataType;
-import org.apache.spark.sql.types.DateType;
-import org.apache.spark.sql.types.DecimalType;
-import org.apache.spark.sql.types.DoubleType;
-import org.apache.spark.sql.types.FloatType;
-import org.apache.spark.sql.types.IntegerType;
-import org.apache.spark.sql.types.LongType;
-import org.apache.spark.sql.types.MapType;
-import org.apache.spark.sql.types.NullType;
-import org.apache.spark.sql.types.ShortType;
-import org.apache.spark.sql.types.StringType;
-import org.apache.spark.sql.types.StructField;
-import org.apache.spark.sql.types.StructType;
-import org.apache.spark.sql.types.TimestampNTZType;
-import org.apache.spark.sql.types.TimestampType;
-import org.apache.spark.sql.types.VarcharType;
-import org.apache.spark.sql.types.VariantType;
-
-class SparkTypeToType extends SparkTypeVisitor<Type> {
+
+public class DeltaLakeKernelTypeToType {
Review Comment:
This could be package private as well?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
Review Comment:
Unnecessary cast?
##########
delta-lake/src/test/resources/delta/golden/README.md:
##########
@@ -0,0 +1,50 @@
+# Delta Lake Golden Tables
Review Comment:
There is a CI failure due to missing license header. May want to add this to `.rat-excludes`
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeActionsTranslationUtil.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.data.Row;
+import io.delta.kernel.internal.actions.AddFile;
+
+/**
+ * Util class helps to handle json operations for <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">delta action</a>
+ */
+public class DeltaLakeActionsTranslationUtil {
Review Comment:
Please consider making this package-private.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
Review Comment:
Can we avoid the use of the internal `AddFile` class and read fields directly from the `Row` using ordinals defined by the scan file schema?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
Review Comment:
Shouldn't we do a fail fast if we encounter a `remove`? Otherwise in tables with DML, the conversion will cause silent duplicates.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+ } else if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ commitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ }
+ }
+ }
+
+ // TODO Append only now
+ if (!dataFilesToAdd.isEmpty()) {
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ }
+
+ tagCurrentSnapshot(deltaVersion, commitTimestamp, transaction);
Review Comment:
If `dataFilesToAdd` is empty, ie line 279 evaluates to false, this line might cause a NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2848466540
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+ } else if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ commitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ }
+ }
+ }
+
+ // TODO Append only now
+ if (!dataFilesToAdd.isEmpty()) {
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ }
+
+ tagCurrentSnapshot(deltaVersion, commitTimestamp, transaction);
Review Comment:
Yes, you are right. It will be no `snapshots` for `empty` tables. I will handle this scenario and add a test for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4766826100
I went though the comments and the code. Let me sum-up the current state:
1. The code was/is ready for next review cycle. Some comments are not resolved because I would like an aka (and resolve) from the author.
2. There is backlog to add more features to the module according to the plan in the PR description.
3. The current version works and functional (with known limitations). I believe it's more pragmatic to get an alignment with community on the current version before increasing the code base with additional logic.
@anoopj
1. The concern regarding usage of internal delta API is actual and valid. As we agreed yearly, I will finish full conversion logic and then collect the full list of the internal API. With the list we will decide if it will be contribution to DeltaKernel or some in-house replacement.
@HonahX
1. I left the comments regarding VACUUM tests. The tests are robust
1. I left the comment regarding VACUUM logic handling
1. All data types (including Variant) tests in unit test. I left the comment.
@laskoviymishka
1. Topic about `validate-from snapshot to currentSnapshot() ` - is not completely clear to me. So, it can be some work here.
1. Testing of deletion vectors per version explained (there is integration test).
1. The `blocker` from the big summary comment:
1.1 DV-update - resolved
1.2 Per-ColumnarBatch commits break - fixed
1.3 Unhandled metaData/protocol actions - fixed
1.4 DV test asserts presence - explained
@anoopj @HonahX @laskoviymishka I would love to get next cycle of your review and input. Thank you for the contributions to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4684468134
> @vladislav-sidorovich is this ready to one more round of review?
The initial inputs from you were addressed. But let me double check everything this weekends to avoid double work from you and other reviewers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3466104191
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
Review Comment:
The schema-evolution guard only runs inside this `isCommitInfo(row)` branch — so a version that carries no `commitInfo` row skips the check entirely. Delta doesn't guarantee a `commitInfo` per version, and the schema is fixed once from `initialDeltaSnapshot` (line 194), so an ADD/RENAME/DROP COLUMN landing in a commitInfo-less version would convert silently against the wrong schema and write a truncated Iceberg table.
That's the silent-corruption case I'd want closed before merge. I'd lift the schema-consistency check out of the per-row commitInfo parsing and verify the schema is stable across the whole `[minimalAvailableDeltaVersion, latestDeltaVersion]` range up front — that way the guard doesn't depend on an action Delta may not have written. wdyt?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
Review Comment:
`icebergPropertiesBuilder` is an instance field that accumulates across calls, and `ImmutableMap.Builder.build()` throws on duplicate keys.
`buildTablePropertiesWithDelta` puts the internal props and then `putAll(configuration)`, on top of whatever the user passed via `tableProperties`/`tableProperty`. Any overlap — a user prop that also appears in the Delta config, or a second `execute()` on the same action — throws `IllegalArgumentException`. I'd make this a local `HashMap` inside the build method, populate user props → Delta config → required Iceberg props so later wins, and return `ImmutableMap.copyOf(...)`.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
Review Comment:
The factory is pinned at `partitionId=1, taskId=1`, and `convertDeltaDVsToIcebergDVs` spins up a fresh `BaseDVFileWriter` per AddFile against this same factory.
Since the factory's counter is the only thing varying the filename and each writer starts from the same fixed seed, the Puffin DV files can land on colliding paths and overwrite each other across files — so we can't really claim correct DV conversion without fixing this alongside the DV double-delete above. Sharing one factory so the counter increments across all DV writers is what the code looks like it intended. `OutputFileFactory` is also `Closeable` and this one is never closed, worth handling in the same pass.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
Review Comment:
`return processedDataFiles::size;` hands back an ad-hoc lambda implementing `Result`, not `ImmutableSnapshotDeltaLakeTable.Result`.
`Set::size` is an `int` widened to the interface's `long snapshotDataFilesCount()`, and the lambda has no `equals`/`hashCode`/`toString`, so it diverges from the Immutables contract the reference impl returns. I'd build it explicitly:
```java
return ImmutableSnapshotDeltaLakeTable.Result.builder()
.snapshotDataFilesCount(processedDataFiles.size())
.build();
```
Worth noting the count also includes removed files (`processedDataFiles.add` runs on both branches), whereas the old impl counted only added files — so the metric semantics drift too. Fix the return type and this one's good to land.
##########
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.ContentFileUtil;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+public class TestBaseSnapshotDeltaLakeKernelTableAction {
+ private static final String DELTA_GOLDEN_TABLES_ROOT = "delta/golden/";
+
+ @TempDir private File sourceDeltaFolder;
+ @TempDir private File icebergWarehouseFolder;
+ private String sourceTableLocation;
+ private final Configuration testHadoopConf = new Configuration();
+ private String newTableLocation;
+ private Catalog testCatalog;
+
+ @BeforeEach
+ public void before() throws IOException {
+ sourceTableLocation = sourceDeltaFolder.toURI().toString();
+
+ Path icebergTablePath = icebergWarehouseFolder.toPath().resolve("iceberg_table");
+ newTableLocation = icebergTablePath.toString();
+ Files.createDirectories(icebergTablePath);
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, icebergWarehouseFolder.toString());
+
+ HadoopCatalog icebergCatalog = new HadoopCatalog();
+ icebergCatalog.setConf(new Configuration());
+ icebergCatalog.initialize("hadoop_catalog", properties);
+ testCatalog = icebergCatalog;
+ }
+
+ @Test
+ public void testRequiredTableIdentifier() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .icebergCatalog(testCatalog)
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredIcebergCatalog() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredDeltaLakeConfiguration() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Make sure to configure the action with a valid deltaLakeConfiguration");
+ }
+
+ @Test
+ public void testDeltaTableColumnMappingFeatureDisabled() throws Exception {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Transaction txn =
+ createEmptyDeltaTableTransaction(engine)
+ .withTableProperties(engine, Map.of("delta.columnMapping.mode", "name"))
+ .build(engine);
+ txn.commit(engine, CloseableIterable.emptyIterable());
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act & check
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Conversion of Delta Lake tables with columnMapping feature is not supported.");
+ }
+
+ @Test
+ public void testEmptyTableConversion() {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ createEmptyDeltaTableTransaction(engine)
+ .build(engine)
+ .commit(engine, CloseableIterable.emptyIterable());
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+
+ org.apache.iceberg.Table table = testCatalog.loadTable(icebergTable);
+ assertThat(((BaseTable) table).operations().current().formatVersion()).isEqualTo(3);
+ }
+
+ @Test
+ public void testTableCreated() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+ }
+
+ @Test
+ public void testDefaultTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("original_location")).isEqualTo(sourceTableLocation);
+ assertThat(properties.get("snapshot_source")).isEqualTo("delta");
+ assertThat(properties.containsKey(TableProperties.DEFAULT_NAME_MAPPING)).isTrue();
+ }
+
+ @Test
+ public void testCustomTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation)
+ .tableProperty("custom_prop_1", "custom val 1")
+ .tableProperty("custom_prop_2", "custom val 2")
+ .tableProperties(
+ ImmutableMap.of(
+ "custom_map_prop1", "val 1",
+ "custom_map_prop2", "val 2"));
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("custom_prop_1")).isEqualTo("custom val 1");
+ assertThat(properties.get("custom_prop_2")).isEqualTo("custom val 2");
+ assertThat(properties.get("custom_map_prop1")).isEqualTo("val 1");
+ assertThat(properties.get("custom_map_prop2")).isEqualTo("val 2");
+ }
+
+ @Test
+ void testDeltaTableNotExist() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Delta Lake table does not exist at the given location: %s", sourceTableLocation);
+ }
+
+ @Test
+ public void testDeltaTableDVSupported() throws Exception {
+ loadDeltaLakeGoldenTable("dv-partitioned-with-checkpoint");
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Table deltaTable = Table.forPath(engine, sourceTableLocation);
+
+ io.delta.kernel.internal.SnapshotImpl deltaSnapshot =
+ (io.delta.kernel.internal.SnapshotImpl) deltaTable.getLatestSnapshot(engine);
+ assertThat(
+ deltaSnapshot
+ .getMetadata()
+ .getConfiguration()
+ .getOrDefault("delta.enableDeletionVectors", "false"))
+ .isEqualTo("true");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act
+ testAction.execute();
+
+ org.apache.iceberg.Table icebergTable = testCatalog.loadTable(icebergTableIdentifier);
+ try (org.apache.iceberg.io.CloseableIterable<org.apache.iceberg.FileScanTask> tasks =
+ icebergTable.newScan().planFiles()) {
+ boolean hasDeleteFiles = false;
+ for (org.apache.iceberg.FileScanTask task : tasks) {
+ for (org.apache.iceberg.DeleteFile deleteFile : task.deletes()) {
+ assertThat(ContentFileUtil.isDV(deleteFile)).isTrue();
+ hasDeleteFiles = true;
+ }
+ }
+ assertThat(hasDeleteFiles).isTrue();
+ }
+ }
+
+ static List<String> deltaGoldenTableNames() throws Exception {
+ Path rootPath =
+ Paths.get(
+ TestBaseSnapshotDeltaLakeKernelTableAction.class
+ .getClassLoader()
+ .getResource(DELTA_GOLDEN_TABLES_ROOT)
+ .toURI());
+
+ try (Stream<Path> stream = Files.list(rootPath)) {
+ return stream
+ .filter(Files::isDirectory)
+ .map(path -> path.getFileName().toString())
+ .collect(Collectors.toList());
+ }
+ }
+
+ @ParameterizedTest
+ @MethodSource("deltaGoldenTableNames")
+ void goldenDeltaTableConversion(String deltaTableName) throws Exception {
+ loadDeltaLakeGoldenTable(deltaTableName);
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ testAction.execute();
Review Comment:
`goldenDeltaTableConversion` only asserts that `execute()` doesn't throw — every golden table passes as long as conversion completes, regardless of whether the result is correct.
For an experimental path that's especially risky, because green CI then implies coverage that isn't actually there. I'd either load the resulting Iceberg table and assert schema, row count, and partitioning against the source, or mark it `@Disabled("follow-up #<issue>")` so it's honest about what it does and doesn't verify. This is the natural place to catch a silently-wrong DV or Remove conversion.
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
+ } else if (!dataFilesToAdd.isEmpty() && dataFilesToRemove.isEmpty()) {
+ // Append only
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ } else if (dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Delete only
+ DeleteFiles deleteFiles = transaction.newDelete();
+ dataFilesToRemove.forEach(deleteFiles::deleteFile);
+ deleteFiles.commit();
+ } else if (!dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Overwrite
+ OverwriteFiles overwriteFiles = transaction.newOverwrite();
+ dataFilesToAdd.forEach(overwriteFiles::addFile);
+ dataFilesToRemove.forEach(overwriteFiles::deleteFile);
+ overwriteFiles.commit();
+ } else {
+ // Tag case
+ transaction.newAppend().commit();
+ }
+ }
+
+ private DataFile buildDataFileFromAddDeltaAction(AddFile addFile, Transaction transaction) {
+ String path = addFile.getPath();
+ long dataFileSize = addFile.getSize();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
+ throw new NotFoundException(
+ "File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage",
+ fullFilePath, deltaTableLocation);
+ }
+
+ MetricsConfig metricsConfig = MetricsConfig.forTable(transaction.table());
+ String nameMappingString =
+ transaction.table().properties().get(TableProperties.DEFAULT_NAME_MAPPING);
+ NameMapping nameMapping =
+ nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ // TODO read metrics from Delta log to avoid data flies read
+ Metrics metrics = ParquetUtil.fileMetrics(inputDataFile, metricsConfig, nameMapping);
+
+ Map<String, String> partitionValues = VectorUtils.toJavaMap(addFile.getPartitionValues());
+ PartitionSpec partitionSpec = transaction.table().spec();
+ List<String> partitionValueList =
+ partitionSpec.fields().stream()
+ .map(PartitionField::name)
+ .map(partitionValues::get)
+ .collect(Collectors.toList());
+
+ return DataFiles.builder(partitionSpec)
+ .withPath(fullFilePath)
+ .withFormat(FileFormat.PARQUET) // Delta supports only parquet datafiles
+ .withFileSizeInBytes(dataFileSize)
+ .withMetrics(metrics)
+ .withPartitionValues(partitionValueList)
+ .build();
+ }
+
+ private DataFile buildDataFileFromRemoveDeltaAction(
+ RemoveFile removeFile, Transaction transaction) {
+ String path = removeFile.getPath();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
Review Comment:
I'm fine deferring the real VACUUM fix — building the removal `DataFile` from the Remove action's own path/size/partition values with no `exists()` or `fileMetrics` read, which lines up with your M3.2 (AddFile/Remove stats conversion). What I'd want for this PR is honesty rather than the full feature.
Two things: this should fail with a clear message — something like "data file removed by VACUUM is not yet supported, see #<issue>" — instead of surfacing a raw `NotFoundException` that reads like a corruption bug. And the PR description should drop the "Support Delta VACUUM scenario" claim until that path actually works, so green CI doesn't imply a capability we don't have yet.
Side note for whoever picks up the follow-up: the metrics read here (`ParquetUtil.fileMetrics`) is also pure waste for a tombstone — a removal `DataFile` only needs path, format, size, and partition values. Folds naturally into the same change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4262134613
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
Review Comment:
This is a table feature, so should be able to check metadata and protocol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6160298966
Thanks for chiming in @anoopj, I think you hit some of the same points I've been thinking about.
1. Iceberg V4 and Delta V5(?) will likely unified the on disk metadata representation 🥳 which makes metadata translation obsolete.
2. UniForm / XTable(?) might handle the translation fairly well already, so we might not need to duplicate the effort here.
That being said, i can still see a place for `iceberg-delta-lake` as a migration path for existing Delta tables into the current Iceberg versions (v1, v2, v3), especially for folks who can't or don't want to enable UniForm. @vladislav-sidorovich has put a lot of time and effort into this PR over the past several months, and i want to make sure that work lands somewhere it'll be used and maintained.
As i see it, there are 2 paths forward: either we gather signal from the community that the module is useful and agree on its scope together, or we sunset it here in favor of moving this logic to another project.
The devlist is a better place to move these discussions. As for this PR, I'm happy to approve after we get rid of the unnecessary test files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6167097315
Thanks @kevinjqliu and @anoopj for the thoughtful feedback and for being open to getting this PR merged!
Test files. Let's discuss it in the thread with your comment to align the state we need to get this PR ready to merge.
Regarding the broader question of whether `iceberg-delta-lake` belongs in the Iceberg repository and the concrete use cases driving it, I’d love to share why we are investing in this module and why it fills a critical gap that UniForm and XTable don't cover:
### 1. Pull-Based Migration (Iceberg-native) vs. Push-Based Interoperability (UniForm)
@kevinjqliu raised a great point that this module spends a lot of effort understanding Delta protocol/Kernel APIs. However, from a user and operational standpoint, **where the tool lives dictates how a migration is executed**:
* **Why users can't always use UniForm:** UniForm is a *push-based* interoperability feature on the Delta side—it requires write access to alter the source Delta table properties, upgrade the table protocol, and trigger metadata generation from a Delta engine. In many real-world enterprise migrations we see, the data engineering team moving to Iceberg either **cannot modify legacy production Delta tables** (due to strict governance or freeze policies) or only has read access to the underlying object storage.
* **Why XTable isn't a drop-in replacement:** Apache XTable (Incubating) is an ambitious omni-directional sync tool, which brings additional operational overhead and is still in incubation. (as of today, it has gaps that block real-world usage in migration scenarios)
* **The role of `iceberg-delta-lake`:** Just as Iceberg provides native migration utilities for Hive and raw Parquet tables (`migrate`, `snapshot`, `add_files`), having a **pull-based, zero-copy migration module natively in Iceberg** allows users to convert Delta tables into native Iceberg v2/v3 tables (including translating Delta DVs to Iceberg Puffin DVs and preserving snapshot history) purely from the Iceberg side without touching the source Delta environment.
### 2. Bridging the Gap to Iceberg v4
@anoopj's point about Iceberg v4 and Delta eventually sharing a unified adaptive metadata tree on disk is super exciting and will be a huge win for the ecosystem long-term!
At the same time, we have strong, immediate demand from customers today who are migrating existing Delta tables (Reader v1–v3 / Writer v1–v7) into Iceberg v2 and v3. This module serves as the practical migration path for those production workloads today. If v4 eventually makes translation obsolete down the road, sunsetting it at that point will be a natural evolution.
### 3. Real-World Migration Complexity & Planned Spark Procedure
Migrating production tables is rarely a single-step metadata swap—it involves catalog registration, path handling, and incremental catch-up during dual-write windows (I recently covered some of these real-world nuances in an article on [migrating tables to GCS and BigLake Iceberg REST Catalog](https://medium.com/google-cloud/migrating-apache-iceberg-tables-to-gcs-and-biglake-iceberg-rest-catalog-c68399d7a03d)). For Delta -> Iceberg, customers currently lack a convenient, native tool.
Once this foundation is merged, my immediate goals are:
1. **Add a Spark Procedure** (similar to [`add_files`](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) / `snapshot` for Hive migrations) so users can run Delta-to-Iceberg migrations directly via Spark SQL.
An example I have in mind looks like this: (of course it's the next step and we will discuss it separately)
```
CALL spark_catalog.system.confert_to_iceberg(
table_type => 'delta',
source_location => 'path/to/table',
...
);
```
2. **Publish a comprehensive How-To Guide** walking the community through end-to-end Delta-to-Iceberg migration scenarios.
### 4. Commitment to Ownership & Maintenance
To address the concern about the module going inactive like it did in 2023: **I am committed to actively maintaining and supporting this module.** We have a structured backlog tracked in [#17146](https://github.com/apache/iceberg/issues/17146) and our [Development Plan Doc](https://docs.google.com/document/d/1eWH7N_9Mo2b1cDm1jFM9R7I16iYa5xZUvtU_9eLozjA/edit?tab=t.0#heading=h.oc0qb4rn6qup) (covering removing internal Delta Kernel usages, incremental sync, column mapping, and schema evolution).
I'm also happy to chime in on the `dev@` mailing list thread so we have this context recorded for the wider community. I'll push the commit removing the extra test files shortly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143783630
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
+ } else if (!dataFilesToAdd.isEmpty() && dataFilesToRemove.isEmpty()) {
+ // Append only
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ } else if (dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Delete only
+ DeleteFiles deleteFiles = transaction.newDelete();
+ dataFilesToRemove.forEach(deleteFiles::deleteFile);
+ deleteFiles.commit();
+ } else if (!dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Overwrite
+ OverwriteFiles overwriteFiles = transaction.newOverwrite();
+ dataFilesToAdd.forEach(overwriteFiles::addFile);
+ dataFilesToRemove.forEach(overwriteFiles::deleteFile);
+ overwriteFiles.commit();
+ } else {
+ // Tag case
+ transaction.newAppend().commit();
+ }
+ }
+
+ private DataFile buildDataFileFromAddDeltaAction(AddFile addFile, Transaction transaction) {
Review Comment:
Yes, it was an initial implementation. The trade-off here is to support additional `if s` in the method and the method was a bit tricky.
So, I decided to split it to keep 2 more straightforward methods. But I still have it in mind, let's see how the logic will increase with other actions...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143955454
##########
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.util.ContentFileUtil;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+public class TestBaseSnapshotDeltaLakeKernelTableAction {
+ private static final String DELTA_GOLDEN_TABLES_ROOT = "delta/golden/";
+
+ @TempDir private File sourceDeltaFolder;
+ @TempDir private File icebergWarehouseFolder;
+ private String sourceTableLocation;
+ private final Configuration testHadoopConf = new Configuration();
+ private String newTableLocation;
+ private Catalog testCatalog;
+
+ @BeforeEach
+ public void before() throws IOException {
+ sourceTableLocation = sourceDeltaFolder.toURI().toString();
+
+ Path icebergTablePath = icebergWarehouseFolder.toPath().resolve("iceberg_table");
+ newTableLocation = icebergTablePath.toString();
+ Files.createDirectories(icebergTablePath);
+
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, icebergWarehouseFolder.toString());
+
+ HadoopCatalog icebergCatalog = new HadoopCatalog();
+ icebergCatalog.setConf(new Configuration());
+ icebergCatalog.initialize("hadoop_catalog", properties);
+ testCatalog = icebergCatalog;
+ }
+
+ @Test
+ public void testRequiredTableIdentifier() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .icebergCatalog(testCatalog)
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredIcebergCatalog() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ }
+
+ @Test
+ public void testRequiredDeltaLakeConfiguration() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Make sure to configure the action with a valid deltaLakeConfiguration");
+ }
+
+ @Test
+ public void testDeltaTableColumnMappingFeatureDisabled() throws Exception {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ Transaction txn =
+ createEmptyDeltaTableTransaction(engine)
+ .withTableProperties(engine, Map.of("delta.columnMapping.mode", "name"))
+ .build(engine);
+ txn.commit(engine, CloseableIterable.emptyIterable());
+
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("iceberg_table"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ // Act & check
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessage("Conversion of Delta Lake tables with columnMapping feature is not supported.");
+ }
+
+ @Test
+ public void testEmptyTableConversion() {
+ Engine engine = DefaultEngine.create(testHadoopConf);
+ createEmptyDeltaTableTransaction(engine)
+ .build(engine)
+ .commit(engine, CloseableIterable.emptyIterable());
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+
+ org.apache.iceberg.Table table = testCatalog.loadTable(icebergTable);
+ assertThat(((BaseTable) table).operations().current().formatVersion()).isEqualTo(3);
+ }
+
+ @Test
+ public void testTableCreated() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTable = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTable)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTable)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTable)).isTrue();
+ }
+
+ @Test
+ public void testDefaultTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("original_location")).isEqualTo(sourceTableLocation);
+ assertThat(properties.get("snapshot_source")).isEqualTo("delta");
+ assertThat(properties.containsKey(TableProperties.DEFAULT_NAME_MAPPING)).isTrue();
+ }
+
+ @Test
+ public void testCustomTableProperties() throws Exception {
+ loadDeltaLakeGoldenTable("basic-decimal-table");
+
+ TableIdentifier icebergTableIdentifier = TableIdentifier.of("iceberg_table");
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(icebergTableIdentifier)
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation)
+ .tableProperty("custom_prop_1", "custom val 1")
+ .tableProperty("custom_prop_2", "custom val 2")
+ .tableProperties(
+ ImmutableMap.of(
+ "custom_map_prop1", "val 1",
+ "custom_map_prop2", "val 2"));
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isFalse();
+
+ testAction.execute();
+
+ assertThat(testCatalog.tableExists(icebergTableIdentifier)).isTrue();
+
+ Map<String, String> properties = testCatalog.loadTable(icebergTableIdentifier).properties();
+ assertThat(properties.get("custom_prop_1")).isEqualTo("custom val 1");
+ assertThat(properties.get("custom_prop_2")).isEqualTo("custom val 2");
+ assertThat(properties.get("custom_map_prop1")).isEqualTo("val 1");
+ assertThat(properties.get("custom_map_prop2")).isEqualTo("val 2");
+ }
+
+ @Test
+ void testDeltaTableNotExist() {
+ SnapshotDeltaLakeTable testAction =
+ new BaseSnapshotDeltaLakeKernelTableAction(sourceTableLocation)
+ .as(TableIdentifier.of("test", "test"))
+ .deltaLakeConfiguration(testHadoopConf)
+ .icebergCatalog(testCatalog)
+ .tableLocation(newTableLocation);
+
+ assertThatThrownBy(testAction::execute)
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(
+ "Delta Lake table does not exist at the given location: %s", sourceTableLocation);
+ }
+
+ @Test
+ public void testDeltaTableDVSupported() throws Exception {
Review Comment:
This unit-test verify the conversion flow and some execution conditions. Additionally it's verify what Delta table features supported and not supported.
Testing for table's data done in integration test org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#testConversionWithDeletionVectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3467892645
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
Review Comment:
Similar to what @anoopj pointed out. Those impl classes to some extend belong to internal classes whose api might change or break
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4107705778
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion,
+ Iterators.transform(changes, batch -> () -> batch),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(currDeltaVersion, commitTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = deltaLatestSnapshot.getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ Long deltaVersion,
+ Iterator<Supplier<ColumnarBatch>> changes,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ while (changes.hasNext()) {
+ Supplier<ColumnarBatch> container = changes.next();
+ ColumnarBatch columnarBatch = container.get();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp =
+ commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+
+ assertSupportedDeltaOperation(deltaVersion, commitInfo);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3571285634
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
You are right, I reproduced the edge case scenario.
Test:
```
String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
String sourceTableLocation = sourceLocation.toURI().toString();
writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
```
I explicitly added it here https://github.com/apache/iceberg/issues/17146
I will handle this edge case as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3572883955
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
Review Comment:
Again, plz don't use Table API but TableManager
https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/TableManager.java
The previous one is kind of deprecated as it is not supported catalog managed table for Delta https://delta.io/blog/2026-02-02-delta-catalog-managed-tables
Delta is also moving to catalog oriented. so using Table API will end up Kernel unable to recognize catalog managed commits(staged comment that is not yet backfilled) causing ether fail query or data lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4262092157
##########
dev/.rat-excludes:
##########
@@ -29,4 +29,5 @@ sitemap.xml
.python-version
**/*_index.md
**/.venv/**
+**/_delta_log/_last_checkpoint
Review Comment:
i think this means that we are potentially packaging `_delta_log/` in the source distribution. i think its better to not do that. we can either generate the golden tables at runtime for testing or exclude them from packaging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6152231331
Thanks @vladislav-sidorovich for driving this, and to everyone who's put in multiple rounds of review here.
I want to zoom out a bit on the direction of `iceberg-delta-lake` and whether it makes sense for it to live in the Iceberg project. Most of what this module does is understand Delta: its log, protocol features, and DV format. The Iceberg side is just writing a table. Keeping it correct means tracking Delta's protocol and Delta Kernel's APIs, which seems like a better fit for a project that already owns that knowledge.
We've touched on this a couple of times on the devlist:
- In ["Inactive modules?"](https://lists.apache.org/thread/fv0hfyjo807yp1vv38qmf7qgprnssnmd), the module came up as a deprecation candidate since there hadn't been meaningful activity since 2023. It was kept based on plans to upgrade it, not a broader decision about whether it belongs here.
- In [the devlist thread for this PR](https://lists.apache.org/thread/z48hr4m28xh6k0ptzrhl52kyxc9r3vrq), the question of sunsetting it came up again, since Delta UniForm can already generate Iceberg metadata (incrementally, with column mapping support) and IcebergCompatV3 should lift the DV limitation. That thread didn't get much wider input.
So i don't think we've really settled this as a community. There are other natural homes for this: Delta itself (UniForm, or contributing this converter there, next to Delta Kernel) or a cross-format project like XTable. Given that, i think it's worth reevaluating whether there's still a benefit to having this module in Iceberg at all.
If we do keep it here, do we have concrete use cases from the community that need it? Would be great to hear from folks who are using this today or planning to.
To be clear, i'm not trying to block this PR, and we can keep iterating on it here. I just think we should be deliberate as a community about whether this belongs in Iceberg before we build more on top of it. Happy to help facilitate the discussion, i can either bump the devlist threads above or start a new thread so we can get more input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6260120636
I agree with the points made by @vladislav-sidorovich and @kevinjqliu . For short term, I recommend merging this PR - a lot of work went into it and it is clearly an improvement over the current Delta Lake code in Iceberg. We can discuss the long-term future of the module in the list - I think it will end up being a very thin translation layer of non-content metadata in the post-v4 world.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2871936776
##########
delta-lake/src/test/resources/delta/golden/README.md:
##########
@@ -0,0 +1,50 @@
+# Delta Lake Golden Tables
Review Comment:
The CI passed, so I will resolve this conversation. Thank you for the input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3468908945
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
Review Comment:
Same here -- use https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/TableManager.java#L76
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3469571019
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeletionVectorConverter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.internal.actions.DeletionVectorDescriptor;
+import io.delta.kernel.internal.deletionvectors.DeletionVectorUtils;
+import io.delta.kernel.internal.deletionvectors.RoaringBitmapArray;
+import io.delta.kernel.internal.util.Tuple2;
Review Comment:
Let's define this in Iceberg or use java native one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3196487820
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
Review Comment:
Talk through offline, those DV-s are harmless and we can safely keep them dangling here, no need to have any action. Closing this thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3139968476
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
Review Comment:
In the old implementation, the [commitInitialDeltaSnapshotToIcebergTransaction](https://github.com/apache/iceberg/blob/57b7c2bd564e5dfa8ccb7cb9e6386e7e942bac0f/delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java#L241), will first to find the actual earliest reconstructable version that have all the data files available. That could help reduce the conversion failure due to VACUUM cleaned up files and breaks time travel ability. What's the design choice behind not using that in the new impl?
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
+ super(ICEBERG_CATALOG_NAME, SparkCatalog.class.getName(), BASE_SPARK_CONFIG);
+ spark.conf().set("spark.sql.catalog." + DEFAULT_SPARK_CATALOG, DeltaCatalog.class.getName());
+ }
+
+ @BeforeAll
+ public static void beforeClass() {
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", NAMESPACE));
+ StructType schema =
+ new StructType(
+ new StructField[] {
+ new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+ new StructField("created_at", DataTypes.TimestampType, true, Metadata.empty()),
+ new StructField("event_name", DataTypes.StringType, true, Metadata.empty()),
+ new StructField("is_active", DataTypes.BooleanType, true, Metadata.empty()),
+ new StructField("price", DataTypes.DoubleType, true, Metadata.empty())
+ });
+ List<Row> data =
+ Arrays.asList(
+ RowFactory.create(1, Timestamp.valueOf("2025-01-01 10:00:00"), "Signup", true, 0.00),
+ RowFactory.create(
+ 2, Timestamp.valueOf("2025-01-02 14:30:00"), "Purchase", true, 199.99),
+ RowFactory.create(
+ 3, Timestamp.valueOf("2025-01-03 09:15:00"), "Deactivation", false, 0.00),
+ RowFactory.create(4, Timestamp.valueOf("2025-01-03 09:16:00"), "Refund", false, 0.00),
+ RowFactory.create(5, Timestamp.valueOf("2025-01-03 09:17:00"), "Refund", false, 0.00));
+ genericDataFrame = spark.createDataFrame(data, schema);
+ }
+
+ @AfterAll
+ public static void afterClass() {
+ spark.sql(String.format("DROP DATABASE IF EXISTS %s CASCADE", NAMESPACE));
+ }
+
+ @Test
+ public void testBasicPartitionedInsertsOnly() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "partitioned_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (12, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_partitioned_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testInsertUpdateDeleteSqls() {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "crud_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id=3;");
+ spark.sql("UPDATE " + sourceTable + " SET id=3 WHERE id=1;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+
+ String newTableIdentifier = toFullTableName(ICEBERG_CATALOG_NAME, "iceberg_crud_table");
+
+ // Act
+ SnapshotDeltaLakeTable conversionAction =
+ DeltaLakeToIcebergMigrationSparkIntegration.snapshotDeltaLakeKernelTable(
+ spark, newTableIdentifier, sourceTableLocation);
+ SnapshotDeltaLakeTable.Result result = conversionAction.execute();
+
+ // Assert
+ checkLatestSnapshotIntegrity(sourceTable, newTableIdentifier);
+ checkTagContentAndOrder(sourceTable, sourceTableLocation, newTableIdentifier, 0);
+ checkIcebergTableLocation(newTableIdentifier, sourceTableLocation);
+ }
+
+ @Test
+ public void testConversionAfterVacuum() throws IOException {
+ String sourceTable = toFullTableName(DEFAULT_SPARK_CATALOG, "vacuumed_table");
+ String sourceTableLocation = sourceLocation.toURI().toString();
+
+ writeDeltaTable(genericDataFrame, sourceTable, sourceTableLocation, "id");
+ for (int i = 0; i < 5; i++) {
+ spark.sql(
+ "UPDATE "
+ + sourceTable
+ + " SET price="
+ + ThreadLocalRandom.current().nextDouble(1000)
+ + " where id="
+ + i
+ + ";");
+ }
+ spark.sql("UPDATE " + sourceTable + " SET created_at=current_date() ;");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (10, current_date(), null, null, null);");
+ spark.sql("INSERT INTO " + sourceTable + " VALUES (11, current_date(), null, null, null);");
+ spark.sql("DELETE FROM " + sourceTable + " WHERE id>=10;");
+ spark.sql("VACUUM " + sourceTable + " RETAIN 0 HOURS");
+ spark.sql(
+ "INSERT INTO " + sourceTable + " VALUES (12, current_date(), 'after_vacuum', null, null);");
+ spark.sql("UPDATE " + sourceTable + " SET id=13 WHERE id=5;");
+
+ // Checkpoint generated. Simulate logs clean-up
+ assertThat(deleteDeltaLogFile("00000000000000000000.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000001.json")).isTrue();
+ assertThat(deleteDeltaLogFile("00000000000000000002.json")).isTrue();
Review Comment:
Does the log clean-up always to happen with the VACUUM? Will there be case where the old logs are not cleaned up and our conversion fails because of the missing data file (deleted by VACUUM)?
Seems this indicate some requirement for conversion to work when VACUUM is called on the source table, could you please elaborate on the scope of supporting vacuum in this PR? Thanks!
##########
delta-lake/src/test/resources/delta/golden/README.md:
##########
@@ -0,0 +1,69 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one
+ - or more contributor license agreements. See the NOTICE file
+ - distributed with this work for additional information
+ - regarding copyright ownership. The ASF licenses this file
+ - to you under the Apache License, Version 2.0 (the
+ - "License"); you may not use this file except in compliance
+ - with the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing,
+ - software distributed under the License is distributed on an
+ - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ - KIND, either express or implied. See the License for the
+ - specific language governing permissions and limitations
+ - under the License.
+ -->
+
+# Delta Lake Golden Tables
+
+## Overview
+This directory contains the **Delta Lake Golden Tables**, a set of reference data tables used to verify compliance with the Delta Lake protocol.
+
+These tables cover various features of the Delta Lake specification (e.g., different protocol versions, partition strategies, checkpoints, and data types). They act as a "compliance suite" to ensure that this project's reader/writer implementation can correctly handle data produced by the official Delta Lake reference implementation.
+
+## Origin & Attribution
+These resources were sourced directly from the open-source Delta Lake project.
+
+* **Source Repository:** [delta-io/delta](https://github.com/delta-io/delta)
+* **Module Path:** `connectors/golden-tables/src`
+* **License:** Apache License 2.0 (See [LICENSE](https://github.com/delta-io/delta/blob/master/LICENSE.txt) in the project)
+
+## Purpose
+The primary goal of these tables is **integration testing**.
+
+Each folder represents a specific "edge case" or feature configuration in Delta Lake. By running the conversion against
+these tables, we verify that:
+1. **Protocol Versioning:** We respect `minReaderVersion` and `minWriterVersion`.
+2. **Feature Support:** We can read tables with `checkpoints`, `vacuumed` files, `deletion vectors`, or `column mapping`.
+3. **Data Consistency:** We read the correct number of rows and column values as defined by the transaction log (`_delta_log`).
+
+## Table Inventory & Explanations
+
+The tables are typically named after the specific feature they test. Below are explanations of common patterns found in this suite:
Review Comment:
[nit] seems we only include basic-decimal-table and dv-partitioned-with-checkpoint. Could we clean up other not included table names here?
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeKernelTable.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.util.LocationUtil;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.SaveMode;
+import org.apache.spark.sql.connector.catalog.CatalogPlugin;
+import org.apache.spark.sql.delta.catalog.DeltaCatalog;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestSnapshotDeltaLakeKernelTable extends SparkDeltaLakeSnapshotTestBase {
+ private static final String NAMESPACE = "delta_conversion_test_ns";
+ private static final String DEFAULT_SPARK_CATALOG = "spark_catalog";
+ private static final String ICEBERG_CATALOG_NAME = "iceberg_hive";
+ private static final Map<String, String> BASE_SPARK_CONFIG =
+ ImmutableMap.of(
+ "type",
+ "hive",
+ "default-namespace",
+ "default",
+ "parquet-enabled",
+ "true",
+ "cache-enabled",
+ "false" // Spark will delete tables using v1, leaving the cache out of sync
+ );
+
+ private static Dataset<Row> genericDataFrame;
+
+ @TempDir private File sourceLocation;
+ @TempDir private File destinationLocation;
+
+ public TestSnapshotDeltaLakeKernelTable() {
Review Comment:
Shall we add tests for unpartitioned table and a table with possible all data types? Also would be good if we have some check similar to `checkDataFilePathsIntegrity` to verify that data file path is absolute and matches those in delta table.
##########
build.gradle:
##########
@@ -638,6 +638,8 @@ project(':iceberg-delta-lake') {
compileOnly libs.immutables.value
compileOnly "io.delta:delta-standalone_${scalaVersion}:${libs.versions.delta.standalone.get()}"
+ compileOnly "io.delta:delta-kernel-api:4.0.0"
Review Comment:
[nit], Could we add the version number to https://github.com/apache/iceberg/blob/main/gradle/libs.versions.toml ?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeletionVectorConverter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.internal.actions.DeletionVectorDescriptor;
+import io.delta.kernel.internal.deletionvectors.DeletionVectorUtils;
+import io.delta.kernel.internal.deletionvectors.RoaringBitmapArray;
+import io.delta.kernel.internal.util.Tuple2;
+
+public class DeletionVectorConverter {
Review Comment:
Could this also be package-private?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java:
##########
@@ -450,11 +448,12 @@ private void tagCurrentSnapshot(long deltaVersion, Transaction transaction) {
*/
private static String getFullFilePath(String path, String tableRoot) {
URI dataFileUri = URI.create(path);
- String decodedPath = URLDecoder.decode(path, StandardCharsets.UTF_8);
if (dataFileUri.isAbsolute()) {
- return decodedPath;
+ return dataFileUri.getScheme().equalsIgnoreCase("file") ? dataFileUri.getPath() : path;
} else {
- return tableRoot + File.separator + decodedPath;
+ String decodedPath = dataFileUri.getPath();
+ String separator = tableRoot.contains(":/") ? "/" : File.separator;
+ return tableRoot + (tableRoot.endsWith(separator) ? "" : separator) + decodedPath;
Review Comment:
Is this change necessary?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
Review Comment:
`assertDeltacolumnMappingDisabled` also calls `getLatestDeltaSnapshot` once. Could we consolidate to only call it once every execution?
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+ while (changes.hasNext()) {
+ FilteredColumnarBatch columnarBatch = changes.next();
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch.getData(), transaction, processedDataFiles);
+ }
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ /**
+ * Current implementation uses the schema conversion mapping based on the logical names. Delta
+ * Lake supports three column mapping modes: none, name, id. So, the renames with columnMapping
+ * feature can lead to data corruption.
+ */
+ private void assertDeltaColumnMappingDisabled(String errorMessage) {
+ Map<String, String> configuration = getLatestDeltaSnapshot().getMetadata().getConfiguration();
+ String columnMappingMode = configuration.getOrDefault("delta.columnMapping.mode", "none");
+ if (!"none".equals(columnMappingMode)) {
+ throw new UnsupportedOperationException(errorMessage);
+ }
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long deltaVersion) {
+ Snapshot snapshot =
+ Preconditions.checkNotNull(
+ deltaTable.getSnapshotAsOfVersion(deltaEngine, deltaVersion),
+ "Delta snapshot for version %s is unreachable.",
+ deltaVersion);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ Long batchTimestamp = null;
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ Long commitTimestamp =
+ commitDeltaColumnarBatchToIcebergTransaction(
+ columnarBatch, transaction, processedDataFiles);
+ if (batchTimestamp == null) {
+ batchTimestamp = commitTimestamp;
+ }
+ }
+
+ tagCurrentSnapshot(currDeltaVersion, batchTimestamp, transaction);
+ }
+ }
+ }
+
+ /**
+ * Convert each delta log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private Long commitDeltaColumnarBatchToIcebergTransaction(
+ ColumnarBatch columnarBatch, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+
+ Long originalCommitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ List<DataFile> dataFilesToRemove = Lists.newArrayList();
+ List<DeleteFile> deleteFilesToAdd = Lists.newArrayList();
+
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ originalCommitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ } else if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromAddDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+
+ List<DeleteFile> deleteFiles =
+ convertDeltaDVsToIcebergDVs(transaction.table().spec(), addFile, dataFile);
+
+ deleteFilesToAdd.addAll(deleteFiles);
+ processedDataFiles.add(dataFile.location());
+ } else if (DeltaLakeActionsTranslationUtil.isRemove(row)) {
+ RemoveFile remove = DeltaLakeActionsTranslationUtil.toRemove(row);
+
+ DataFile dataFile = buildDataFileFromRemoveDeltaAction(remove, transaction);
+ dataFilesToRemove.add(dataFile);
+ processedDataFiles.add(dataFile.location());
+ }
+ }
+ }
+
+ // TODO support more actions
+ commitIcebergTransaction(transaction, dataFilesToAdd, dataFilesToRemove, deleteFilesToAdd);
+
+ return originalCommitTimestamp;
+ }
+
+ private List<DeleteFile> convertDeltaDVsToIcebergDVs(
+ PartitionSpec partitionSpec, AddFile addFile, DataFile dataFile) throws IOException {
+ if (addFile.getDeletionVector().isEmpty()) {
+ return List.of();
+ }
+
+ DVFileWriter dvWriter = new BaseDVFileWriter(icebergDVFileFactory, path -> null);
+ try (DVFileWriter closeableWriter = dvWriter) {
+ long[] positions =
+ deletionVectorConverter.readDeltaDVPositions(addFile.getDeletionVector().get());
+ for (long deletedRowIndex : positions) {
+ closeableWriter.delete(
+ dataFile.location(), deletedRowIndex, partitionSpec, dataFile.partition());
+ }
+ }
+
+ return dvWriter.result().deleteFiles();
+ }
+
+ /**
+ * CASES:
+ *
+ * <ol>
+ * <li>Append only
+ * <li>Delete only
+ * <li>Append and Delete => overwrite
+ * <li>RowDelta with deletes.
+ * <li>No Append, No Delete => No data changes, append tag or snapshot.
+ * </ol>
+ */
+ private static void commitIcebergTransaction(
+ Transaction transaction,
+ List<DataFile> dataFilesToAdd,
+ List<DataFile> dataFilesToRemove,
+ List<DeleteFile> deleteFilesToAdd) {
+ if (!deleteFilesToAdd.isEmpty()) {
+ // Row Delta
+ RowDelta rowDelta = transaction.newRowDelta();
+ // Avoid validation for multiple DVs added in transaction
+ // org/apache/iceberg/MergingSnapshotProducer.java:854
+ // since we do the conversion sequentially in a single Iceberg transaction
+ rowDelta.validateFromSnapshot(transaction.table().currentSnapshot().snapshotId());
+
+ dataFilesToAdd.forEach(rowDelta::addRows);
+ dataFilesToRemove.forEach(rowDelta::removeRows);
+ deleteFilesToAdd.forEach(rowDelta::addDeletes);
+ rowDelta.commit();
+ } else if (!dataFilesToAdd.isEmpty() && dataFilesToRemove.isEmpty()) {
+ // Append only
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ } else if (dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Delete only
+ DeleteFiles deleteFiles = transaction.newDelete();
+ dataFilesToRemove.forEach(deleteFiles::deleteFile);
+ deleteFiles.commit();
+ } else if (!dataFilesToAdd.isEmpty() && !dataFilesToRemove.isEmpty()) {
+ // Overwrite
+ OverwriteFiles overwriteFiles = transaction.newOverwrite();
+ dataFilesToAdd.forEach(overwriteFiles::addFile);
+ dataFilesToRemove.forEach(overwriteFiles::deleteFile);
+ overwriteFiles.commit();
+ } else {
+ // Tag case
+ transaction.newAppend().commit();
+ }
+ }
+
+ private DataFile buildDataFileFromAddDeltaAction(AddFile addFile, Transaction transaction) {
Review Comment:
[nit] Most of the logic in this method is the same as `buildDataFileFromRemoveDeltaAction`, could we extract the common "buildDataFile" logic to reduce code duplication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3207486037
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
+
+ DataFile dataFile = buildDataFileFromDeltaAction(addFile, transaction);
+ dataFilesToAdd.add(dataFile);
+ } else if (DeltaLakeActionsTranslationUtil.isCommitInfo(row)) {
+ Row commitInfo = row.getStruct(row.getSchema().indexOf("commitInfo"));
+ commitTimestamp = commitInfo.getLong(commitInfo.getSchema().indexOf("timestamp"));
+ }
+ }
+ }
+
+ // TODO Append only now
+ if (!dataFilesToAdd.isEmpty()) {
+ AppendFiles appendFiles = transaction.newAppend();
+ dataFilesToAdd.forEach(appendFiles::appendFile);
+ appendFiles.commit();
+ }
+
+ tagCurrentSnapshot(deltaVersion, commitTimestamp, transaction);
+
+ return dataFilesToAdd.size();
+ }
+
+ // TODO support more actions
+ private DataFile buildDataFileFromDeltaAction(AddFile addFile, Transaction transaction) {
+ String path = addFile.getPath();
+ long dataFileSize = addFile.getSize();
+ String fullFilePath = getFullFilePath(path, deltaTable.getPath(deltaEngine));
+
+ InputFile inputDataFile = deltaLakeFileIO.newInputFile(fullFilePath);
+ if (!inputDataFile.exists()) {
+ throw new NotFoundException(
+ "File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage",
+ fullFilePath, deltaTableLocation);
+ }
+
+ MetricsConfig metricsConfig = MetricsConfig.forTable(transaction.table());
+ String nameMappingString =
+ transaction.table().properties().get(TableProperties.DEFAULT_NAME_MAPPING);
+ NameMapping nameMapping =
+ nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ Metrics metrics = ParquetUtil.fileMetrics(inputDataFile, metricsConfig, nameMapping);
+
+ Map<String, String> partitionValues = VectorUtils.toJavaMap(addFile.getPartitionValues());
+ PartitionSpec partitionSpec = transaction.table().spec();
+ List<String> partitionValueList =
+ partitionSpec.fields().stream()
+ .map(PartitionField::name)
+ .map(partitionValues::get)
+ .collect(Collectors.toList());
+
+ return DataFiles.builder(partitionSpec)
+ .withPath(fullFilePath)
+ .withFormat(FileFormat.PARQUET) // Delta supports only parquet datafiles
+ .withFileSizeInBytes(dataFileSize)
+ .withMetrics(metrics)
+ .withPartitionValues(partitionValueList)
+ .build();
+ }
+
+ @Nonnull
+ private static Schema convertToIcebergSchema(StructType deltaSchema) {
+ Type converted = new DeltaLakeKernelTypeToType(deltaSchema).convertType();
+ return new Schema(converted.asNestedType().asStructType().fields());
+ }
+
+ private long getEarliestDeltaLog() {
+ try {
+ // "_delta_log" is unmodifiable logs location
+ return DeltaHistoryManager.getEarliestDeltaFile(
+ deltaEngine, new Path(deltaTableLocation, "_delta_log"));
+ } catch (TableNotFoundException e) {
+ throw deltaTableNotFoundException(e);
+ }
+ }
+
+ private SnapshotImpl getLatestDeltaSnapshot() {
+ Snapshot latestSnapshot;
+ try {
+ latestSnapshot = deltaTable.getLatestSnapshot(deltaEngine);
+
+ assertSnapshotImpl(latestSnapshot);
+
+ return (SnapshotImpl) latestSnapshot;
+ } catch (TableNotFoundException e) {
+ throw deltaTableNotFoundException(e);
+ }
+ }
+
+ private PartitionSpec buildPartitionSpec(Schema schema, List<String> partitionNames) {
+ if (partitionNames.isEmpty()) {
+ return PartitionSpec.unpartitioned();
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(schema);
+ for (String partitionName : partitionNames) {
+ builder.identity(partitionName);
+ }
+ return builder.build();
+ }
+
+ private Map<String, String> buildTablePropertiesWithDelta(
+ SnapshotImpl deltaSnapshot, String originalLocation) {
+ icebergPropertiesBuilder.putAll(
+ org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.of(
+ SNAPSHOT_SOURCE_PROP, DELTA_SOURCE_VALUE, ORIGINAL_LOCATION_PROP, originalLocation));
+
+ Map<String, String> configuration = deltaSnapshot.getMetadata().getConfiguration();
+ icebergPropertiesBuilder.putAll(configuration);
+
+ return icebergPropertiesBuilder.build();
+ }
+
+ private void tagCurrentSnapshot(
+ long deltaVersion, Long deltaVersionTimestamp, Transaction transaction) {
+ long currentSnapshotId = transaction.table().currentSnapshot().snapshotId();
+
+ ManageSnapshots manageSnapshots = transaction.manageSnapshots();
+ manageSnapshots.createTag(DELTA_VERSION_TAG_PREFIX + deltaVersion, currentSnapshotId);
+
+ if (deltaVersionTimestamp != null) {
+ manageSnapshots.createTag(
+ DELTA_TIMESTAMP_TAG_PREFIX + deltaVersionTimestamp, currentSnapshotId);
+ }
+ manageSnapshots.commit();
+ }
+
+ private static void assertSnapshotImpl(Snapshot latestSnapshot) {
+ if (!(latestSnapshot instanceof SnapshotImpl)) {
+ throw new IllegalStateException(
+ "Unsupported impl of delta Snapshot: " + latestSnapshot.getClass());
+ }
+ }
+
+ @Nonnull
+ private IllegalArgumentException deltaTableNotFoundException(TableNotFoundException exception) {
+ return new IllegalArgumentException(
+ String.format(
+ "Delta Lake table does not exist at the given location: %s", deltaTableLocation),
+ exception);
+ }
+
+ private static String getFullFilePath(String path, String tableRoot) {
+ URI dataFileUri = URI.create(path);
+ String decodedPath = URLDecoder.decode(path, StandardCharsets.UTF_8);
+ if (dataFileUri.isAbsolute()) {
+ return decodedPath;
+ } else {
+ return tableRoot + File.separator + decodedPath;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4107690649
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
Review Comment:
Supported.
There are 2 tests:
org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#testConversionAfterVacuum
org.apache.iceberg.delta.TestSnapshotDeltaLakeKernelTable#testConversionAfterVacuumNoCheckpoints
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4270274838
##########
dev/.rat-excludes:
##########
@@ -29,4 +29,5 @@ sitemap.xml
.python-version
**/*_index.md
**/.venv/**
+**/_delta_log/_last_checkpoint
Review Comment:
> we are potentially packaging _delta_log/ in the source distribution. ...
No, it's only test usage. But it's good point, I need to clean-up this file.
> I have some concerns with adding 100+ "golden files" into the repo, is it really necessary? can we just generate these?
It's trade-off `Static Golden Files` vs. `Dynamic Generation`.
From my point of view, static files are more reliable for testing scenarios.
Some motivation behind my point of view:
1. The purpose of these golden files is to test protocol conformance across specific. So, it's exactly the case.
2. Since protocol specs are immutable, these files shouldn't change over time.
3. Generating all of these tables dynamically during test runs requires spinning up a full writer engine (like Spark + `delta-spark`) in the middle of unit tests. That adds an extra layer of tooling that can be error-prone.
4. This follows the exact same philosophy Iceberg recently adopted with [`apache/iceberg-verification`](https://github.com/apache/iceberg-verification), where language-neutral, immutable conformance fixtures are used to verify reader implementations rather than having each test generate its own data on the fly.
Unfortunately we can't pull the golden tables as a test-scoped Maven dependency/archive during the build, IMHO it would be an ideal solution. @anoopj what do you think, is it possible to create an artifact with the golden tables?
What are your thoughts on this? Could you share a bit more about your concern so we can find the best approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6154411892
I work on both Delta Lake and Iceberg. I [raised](https://lists.apache.org/thread.html/np5dzr9bjzfyo7ql359hkdthdpw3fm4d) the sunsetting question earlier this year. I don't mind this PR getting merged, but my recommendation is to sunset this module in the long term. This is because UniForm handles the current conversion reasonably well.
Note that metadata translation is expensive, regardless of who does the translation (UniForm or this module). So longer-term Delta Lake is [moving](https://docs.google.com/presentation/d/1WTzXcgHlQZzYz1MiZv_LdV-mZ1vfqJ5rkRXzF0755p4/edit?slide=id.p5#slide=id.p5) towards a unified content metadata with Iceberg v4. The v4 adaptive metadata tree will be used to store the content metadata in Delta. So the metadata representation will be the same on disk, making most translation unnecessary and improving out-of-the-box interoperability between the projects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6407114404
Thanks for the quick turnaround @vladislav-sidorovich! Removing the golden files from the repo is great, but i don't think downloading them at test time is the right approach either. A few concerns:
- if the download fails, the tests are silently skipped, so CI can go green without actually running them
- the build now depends on a GitHub release we don't control. Release verification (`./gradlew build` from the source tarball) shouldn't need network access to a third-party repo
- `delta-incubator/dat` doesn't seem to have a license
- adds a new gradle plugin, writes to `~/.m2` under made-up coordinates, and adds `mavenLocal()` as a dependency source
- nothing to do with the download, but `jackson-databind` is added as an `implementation` dependency and i don't see it used anywhere. is that needed?
Can we generate the tables at runtime instead? The tests only need 2 tables: a basic one can be created with Delta Kernel in the unit tests, and the DV one with `delta-spark` in the integration tests (`testConversionWithDeletionVectors` already does something similar). That way everything is self-contained and we don't need to vendor or download anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2880668533
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, total data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ totalDataFiles);
+ return () -> totalDataFiles;
+ }
+
+ private static void setDefaultNamingMapping(Transaction transaction) {
+ transaction
+ .table()
+ .updateProperties()
+ .set(
+ TableProperties.DEFAULT_NAME_MAPPING,
+ NameMappingParser.toJson(MappingUtil.create(transaction.table().schema())))
+ .commit();
+ }
+
+ private SnapshotImpl getDeltaSnapshotAsOfVersion(long earliestDeltaFile) {
+ Snapshot snapshot = deltaTable.getSnapshotAsOfVersion(deltaEngine, earliestDeltaFile);
+ assertSnapshotImpl(snapshot);
+ return (SnapshotImpl) snapshot;
+ }
+
+ private long convertEachDeltaVersion(
+ long initialDeltaVersion, long latestDeltaVersion, Transaction transaction)
+ throws IOException {
+ LOG.info(
+ "Log for compilation {}, {}, {}", initialDeltaVersion, latestDeltaVersion, transaction);
+
+ long dataFiles = 0;
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
+ deltaEngine,
+ currDeltaVersion,
+ currDeltaVersion,
+ Set.of(DeltaLogActionUtils.DeltaAction.values()))) {
+ while (changes.hasNext()) {
+ ColumnarBatch columnarBatch = changes.next();
+
+ dataFiles +=
+ commitDeltaColumnarBatchToIcebergTransaction(
+ currDeltaVersion, columnarBatch, transaction);
+ }
+ }
+ }
+ return dataFiles;
+ }
+
+ /**
+ * Convert each dela log {@code ColumnarBatch} to Iceberg action and commit to the given {@code
+ * Transaction}. The complete <a
+ * href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Actions">spec</a> of delta
+ * actions. <br>
+ * Supported:
+ * <li>Add
+ *
+ * @return number of added data files
+ */
+ private long commitDeltaColumnarBatchToIcebergTransaction(
+ long deltaVersion, ColumnarBatch columnarBatch, Transaction transaction) throws IOException {
+ // TODO
+ // 1. initial delta version with all the data files
+ // 1.1 data skipping stats
+ // 1.2 DVs support
+ // 2. Delta log to Iceberg history 1 by 1
+ // 3. Delta versions and Delta tags
+ // DeltaLogActionUtils.readCommitFiles(engine, commitFiles, readSchema);
+ // DeltaLogFile
+ // io.delta.kernel.internal.util.FileNames.deltaVersion(io.delta.kernel.internal.fs.Path)
+
+ Long commitTimestamp = null;
+ List<DataFile> dataFilesToAdd = Lists.newArrayList();
+ try (CloseableIterator<Row> rows = columnarBatch.getRows()) {
+ while (rows.hasNext()) {
+ Row row = rows.next();
+ if (DeltaLakeActionsTranslationUtil.isAdd(row)) {
+ AddFile addFile = DeltaLakeActionsTranslationUtil.toAdd(row);
Review Comment:
Yes, I will refactor this part after all the conversion features will be in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2880633137
##########
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeKernelTypeToType.java:
##########
@@ -16,47 +16,74 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.iceberg.spark;
-
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.types.ArrayType;
+import io.delta.kernel.types.BinaryType;
+import io.delta.kernel.types.BooleanType;
+import io.delta.kernel.types.ByteType;
+import io.delta.kernel.types.DataType;
+import io.delta.kernel.types.DateType;
+import io.delta.kernel.types.DecimalType;
+import io.delta.kernel.types.DoubleType;
+import io.delta.kernel.types.FieldMetadata;
+import io.delta.kernel.types.FloatType;
+import io.delta.kernel.types.IntegerType;
+import io.delta.kernel.types.LongType;
+import io.delta.kernel.types.MapType;
+import io.delta.kernel.types.ShortType;
+import io.delta.kernel.types.StringType;
+import io.delta.kernel.types.StructField;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.types.TimestampNTZType;
+import io.delta.kernel.types.TimestampType;
+import io.delta.kernel.types.VariantType;
import java.util.List;
+import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
-import org.apache.spark.sql.types.ArrayType;
-import org.apache.spark.sql.types.BinaryType;
-import org.apache.spark.sql.types.BooleanType;
-import org.apache.spark.sql.types.ByteType;
-import org.apache.spark.sql.types.CharType;
-import org.apache.spark.sql.types.DataType;
-import org.apache.spark.sql.types.DateType;
-import org.apache.spark.sql.types.DecimalType;
-import org.apache.spark.sql.types.DoubleType;
-import org.apache.spark.sql.types.FloatType;
-import org.apache.spark.sql.types.IntegerType;
-import org.apache.spark.sql.types.LongType;
-import org.apache.spark.sql.types.MapType;
-import org.apache.spark.sql.types.NullType;
-import org.apache.spark.sql.types.ShortType;
-import org.apache.spark.sql.types.StringType;
-import org.apache.spark.sql.types.StructField;
-import org.apache.spark.sql.types.StructType;
-import org.apache.spark.sql.types.TimestampNTZType;
-import org.apache.spark.sql.types.TimestampType;
-import org.apache.spark.sql.types.VarcharType;
-import org.apache.spark.sql.types.VariantType;
-
-class SparkTypeToType extends SparkTypeVisitor<Type> {
+
+public class DeltaLakeKernelTypeToType {
Review Comment:
Agree. fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2958661097
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
Review Comment:
Got it, thank you. So, it's tested now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion - Inserts only [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r2855543796
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ long latestDeltaVersion = getLatestDeltaSnapshot().getVersion();
+ long initialDeltaVersion = getEarliestDeltaLog();
+
+ // the initialDeltaVersion used as a lower bound and the actual snapshot can be at bigger
+ // version because of possible concurrent operations. It's ok.
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(initialDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ initialDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+
+ long totalDataFiles;
+ try {
+ totalDataFiles =
+ convertEachDeltaVersion(initialDeltaVersion, latestDeltaVersion, transaction);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
Review Comment:
Done. tnx
##########
delta-lake/src/integration/java/org/apache/iceberg/delta/DeltaLakeToIcebergMigrationSparkIntegration.java:
##########
@@ -60,4 +60,35 @@ static SnapshotDeltaLakeTable snapshotDeltaLakeTable(
.deltaLakeConfiguration(spark.sessionState().newHadoopConf())
.icebergCatalog(Spark3Util.loadIcebergCatalog(spark, catalogAndIdent.catalog().name()));
}
+
+ /**
+ * Example of how to use a {@link SparkSession}, a table identifier and a delta table location to
+ * construct an action for snapshotting the delta table to an iceberg table.
+ *
+ * @param spark a SparkSession with iceberg catalog configured.
+ * @param newTableIdentifier can be both 2 parts and 3 parts identifier, if it is 2 parts, the
+ * default spark catalog will be used
+ * @param deltaTableLocation the location of the delta table
+ * @return an instance of snapshot delta lake table action.
+ */
+ static SnapshotDeltaLakeTable snapshotDeltaLakeKernelTable(
+ SparkSession spark, String newTableIdentifier, String deltaTableLocation) {
+ Preconditions.checkArgument(
+ spark != null, "The SparkSession cannot be null, please provide a valid SparkSession");
+ Preconditions.checkArgument(
+ newTableIdentifier != null,
+ "The table identifier cannot be null, please provide a valid table identifier for the new iceberg table");
+ Preconditions.checkArgument(
+ deltaTableLocation != null,
+ "The delta lake table location cannot be null, please provide a valid location of the delta lake table to be snapshot");
Review Comment:
Done
##########
delta-lake/src/test/resources/delta/golden/README.md:
##########
@@ -0,0 +1,50 @@
+# Delta Lake Golden Tables
Review Comment:
What do you think about adding the Apache 2 license header to this file?
I think it should be ok, because the `README` was created by me for you and other developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143717746
##########
build.gradle:
##########
@@ -638,6 +638,8 @@ project(':iceberg-delta-lake') {
compileOnly libs.immutables.value
compileOnly "io.delta:delta-standalone_${scalaVersion}:${libs.versions.delta.standalone.get()}"
+ compileOnly "io.delta:delta-kernel-api:4.0.0"
Review Comment:
Yep, absolutely. Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3143763703
##########
delta-lake/src/test/resources/delta/golden/README.md:
##########
@@ -0,0 +1,69 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one
+ - or more contributor license agreements. See the NOTICE file
+ - distributed with this work for additional information
+ - regarding copyright ownership. The ASF licenses this file
+ - to you under the Apache License, Version 2.0 (the
+ - "License"); you may not use this file except in compliance
+ - with the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing,
+ - software distributed under the License is distributed on an
+ - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ - KIND, either express or implied. See the License for the
+ - specific language governing permissions and limitations
+ - under the License.
+ -->
+
+# Delta Lake Golden Tables
+
+## Overview
+This directory contains the **Delta Lake Golden Tables**, a set of reference data tables used to verify compliance with the Delta Lake protocol.
+
+These tables cover various features of the Delta Lake specification (e.g., different protocol versions, partition strategies, checkpoints, and data types). They act as a "compliance suite" to ensure that this project's reader/writer implementation can correctly handle data produced by the official Delta Lake reference implementation.
+
+## Origin & Attribution
+These resources were sourced directly from the open-source Delta Lake project.
+
+* **Source Repository:** [delta-io/delta](https://github.com/delta-io/delta)
+* **Module Path:** `connectors/golden-tables/src`
+* **License:** Apache License 2.0 (See [LICENSE](https://github.com/delta-io/delta/blob/master/LICENSE.txt) in the project)
+
+## Purpose
+The primary goal of these tables is **integration testing**.
+
+Each folder represents a specific "edge case" or feature configuration in Delta Lake. By running the conversion against
+these tables, we verify that:
+1. **Protocol Versioning:** We respect `minReaderVersion` and `minWriterVersion`.
+2. **Feature Support:** We can read tables with `checkpoints`, `vacuumed` files, `deletion vectors`, or `column mapping`.
+3. **Data Consistency:** We read the correct number of rows and column values as defined by the transaction log (`_delta_log`).
+
+## Table Inventory & Explanations
+
+The tables are typically named after the specific feature they test. Below are explanations of common patterns found in this suite:
Review Comment:
Not really. Initially I added all the table, but then I removed them from the PR to reduce the PR size. I will add other tables in the follow-up PRs to keep test coverage robust.
The plan described in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3519053210
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
+ }
+
+ private void commitDeltaSnapshotToIcebergTransaction(
+ SnapshotImpl deltaSnapshot, Transaction transaction, Set<String> processedDataFiles)
+ throws IOException {
+ Scan scan = deltaSnapshot.getScanBuilder().build();
+ try (CloseableIterator<FilteredColumnarBatch> changes = scan.getScanFiles(deltaEngine)) {
+
+ commitDeltaColumnarBatchToIcebergTransaction(
+ deltaSnapshot.getVersion(),
+ Iterators.transform(changes, batch -> batch::getData),
+ transaction,
+ processedDataFiles);
+ tagCurrentSnapshot(
+ deltaSnapshot.getVersion(), deltaSnapshot.getTimestamp(deltaEngine), transaction);
+ }
+ }
+
+ private void convertEachDeltaVersion(
+ long initialDeltaVersion,
+ long latestDeltaVersion,
+ Transaction transaction,
+ Set<String> processedDataFiles)
+ throws IOException {
+
+ for (long currDeltaVersion = initialDeltaVersion;
+ currDeltaVersion <= latestDeltaVersion;
+ currDeltaVersion++) {
+ try (CloseableIterator<ColumnarBatch> changes =
+ deltaTable.getChanges(
Review Comment:
Got it thanks. The API you point is marked as `Experimental` and not available in delta-kernel:4.0.0.
Anyway, there is a plan to refactor internal API in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#issuecomment-4833023765
> Thanks for sticking with the review iterations. I went back through the existing threads before adding anything here, so I’m not reopening what’s already settled.
>
> .....
>
> Once the correctness issues are fixed and the follow-ups are tracked, happy to take another pass and approve.
@laskoviymishka Thank you for the review and the summary comment.
1. >`The DV `OutputFileFactory` is fixed at `(1, 1)`...`
The following sequence of action
```
1. Add(file_A, DV_1)
2. Remove(file_A, DV_1)
3. Add(file_A, DV_2)
```
is possible in multiple Delta commits and handled in `io.delta.kernel.internal.replay.ActiveAddFilesIterator` by `UniqueFileActionTuple`. At the same time this sequence is impossible in the scope of single Delta snapshot and handled by `DMLWithDeletionVectorsHelper.scala` for writes, and `ConflictChecker.scala` for reads (see `lazy val addedFilePathToActionMap: Map[String, AddFile]`). The conversion works snapshot by snapshot, so it's correct.
The factory is not closable, but `DVFileWriter dvWriter` is closed in try-with-resource section in `convertDeltaDVsToIcebergDVs`.
2. > execute() returns a method-reference lambda instead ...
Done.
3. > icebergPropertiesBuilder is an instance ImmutableMap.Builder ...
throw on duplicate keys is an expected behavior. It’s never happened in a proper execution but will throw an exception for invalid usage. This strategy also aligns with your previous comments regarding fail with explicit message vs silent corruption.
4. > The golden-table test only checks that execute() doesn’t throw ....
It's actually an interesting topic to discus. My ideas was the following: use unit-tests `TestBaseSnapshotDeltaLakeKernelTableAction` to verify conversion logic like supported / not supported features, valid/invalid parameters, basically logic I wrote for the conversion. Testing the table data is done in the integration tests and moving it to unit-tests will make them much slower. So, I see the following options:
1. Keep golden tables in unit-tests and follow the pattern to throw an exception instead of silent corruption.
2. Move golden tables to integration-tests and verify data
What do you think?
Other topics:
1. VACUUM - I'm not 100% aligned there. I believe VACUUM logic is covered and work properly. Please let me know if I missed anything.
1. `testConversionAfterVacuum` - tests to processes `VACUUM` and logs compaction. VACUUM deletes data files, and log compaction deletes logs before check points. So, in this test we have commits pointing to deleted files but it works properly because conversion support VACUUM and starts from the latest checkpoint.
1. Let me go again through all the comments to double check that I didn't miss anything.
1. I will move the implementation doc to issues as was requested.
1. I will move all the usage of internal Delta API to helper classes to have it in 1 place because it was highlighted by at least 2 reviewers .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3491893471
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,629 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.data.ColumnarBatch;
+import io.delta.kernel.data.FilteredColumnarBatch;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.internal.DeltaHistoryManager;
+import io.delta.kernel.internal.DeltaLogActionUtils;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.TableImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.fs.Path;
+import io.delta.kernel.internal.util.VectorUtils;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterator;
+import java.io.File;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.AppendFiles;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.DeleteFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Metrics;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.OverwriteFiles;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.deletes.BaseDVFileWriter;
+import org.apache.iceberg.deletes.DVFileWriter;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFileFactory;
+import org.apache.iceberg.mapping.MappingUtil;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.parquet.ParquetUtil;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BaseSnapshotDeltaLakeKernelTableAction implements SnapshotDeltaLakeTable {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(BaseSnapshotDeltaLakeKernelTableAction.class);
+
+ private static final String SNAPSHOT_SOURCE_PROP = "snapshot_source";
+ private static final String DELTA_SOURCE_VALUE = "delta";
+ private static final String ORIGINAL_LOCATION_PROP = "original_location";
+ private static final String DELTA_VERSION_TAG_PREFIX = "delta-version-";
+ private static final String DELTA_TIMESTAMP_TAG_PREFIX = "delta-ts-";
+ private static final Set<String> UNSUPPORTED_DELTA_OPERATIONS =
+ Set.of(
+ "ADD COLUMNS",
+ "CHANGE COLUMN",
+ "RENAME COLUMN",
+ "DROP COLUMN",
+ "ADD CONSTRAINT",
+ "SET TBLPROPERTIES"); // The operations will be supported eventually
+
+ private final ImmutableMap.Builder<String, String> icebergPropertiesBuilder =
+ ImmutableMap.builder();
+
+ private final String deltaTableLocation;
+ private Engine deltaEngine;
+ private TableImpl deltaTable;
+ private SnapshotImpl deltaLatestSnapshot;
+
+ private Catalog icebergCatalog;
+ private TableIdentifier newTableIdentifier;
+ private String newTableLocation;
+ private HadoopFileIO deltaLakeFileIO;
+ private DeletionVectorConverter deletionVectorConverter;
+ private OutputFileFactory icebergDVFileFactory;
+ private final Set<Long> deltaTimestampTags = Sets.newHashSet();
+
+ BaseSnapshotDeltaLakeKernelTableAction(String deltaTableLocation) {
+ this.deltaTableLocation = deltaTableLocation;
+ this.newTableLocation = deltaTableLocation;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperties(Map<String, String> properties) {
+ icebergPropertiesBuilder.putAll(properties);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableProperty(String name, String value) {
+ icebergPropertiesBuilder.put(name, value);
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable tableLocation(String location) {
+ newTableLocation = location;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable as(TableIdentifier identifier) {
+ newTableIdentifier = identifier;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable icebergCatalog(Catalog catalog) {
+ icebergCatalog = catalog;
+ return this;
+ }
+
+ @Override
+ public SnapshotDeltaLakeTable deltaLakeConfiguration(Configuration conf) {
+ deltaEngine = DefaultEngine.create(conf);
+ deltaLakeFileIO = new HadoopFileIO(conf);
+ deltaTable = (TableImpl) Table.forPath(deltaEngine, deltaTableLocation);
+ deletionVectorConverter = new DeletionVectorConverter(deltaEngine, deltaTableLocation);
+ return this;
+ }
+
+ @Override
+ public Result execute() {
+ Preconditions.checkArgument(
+ icebergCatalog != null && newTableIdentifier != null,
+ "Iceberg catalog and identifier cannot be null. Make sure to configure the action with a valid Iceberg catalog and identifier.");
+ Preconditions.checkArgument(
+ deltaTable != null && deltaLakeFileIO != null,
+ "Make sure to configure the action with a valid deltaLakeConfiguration");
+
+ loadLatestDeltaSnapshot();
+ assertDeltaColumnMappingDisabled(
+ "Conversion of Delta Lake tables with columnMapping feature is not supported.");
+
+ final long latestDeltaVersion = deltaLatestSnapshot.getVersion();
+ final long minimalAvailableDeltaVersion = getEarliestRecreatableDeltaLog();
+
+ SnapshotImpl initialDeltaSnapshot = getDeltaSnapshotAsOfVersion(minimalAvailableDeltaVersion);
+
+ LOG.info(
+ "Converting Delta Lake table at {} from version {} to version {} into Iceberg table {} ...",
+ deltaTableLocation,
+ minimalAvailableDeltaVersion,
+ latestDeltaVersion,
+ newTableIdentifier);
+
+ Schema icebergSchema = convertToIcebergSchema(initialDeltaSnapshot.getSchema());
+ PartitionSpec partitionSpec =
+ buildPartitionSpec(icebergSchema, initialDeltaSnapshot.getPartitionColumnNames());
+
+ Transaction transaction =
+ icebergCatalog.newCreateTableTransaction(
+ newTableIdentifier,
+ icebergSchema,
+ partitionSpec,
+ newTableLocation,
+ buildTablePropertiesWithDelta(initialDeltaSnapshot, deltaTableLocation));
+ setDefaultNamingMapping(transaction);
+ // Create an initial empty snapshot so currentSnapshot() is never null
+ transaction.newAppend().commit();
+
+ icebergDVFileFactory =
+ OutputFileFactory.builderFor(transaction.table(), 1, 1).format(FileFormat.PUFFIN).build();
+
+ Set<String> processedDataFiles = Sets.newHashSet();
+ try {
+ commitDeltaSnapshotToIcebergTransaction(
+ initialDeltaSnapshot, transaction, processedDataFiles);
+ convertEachDeltaVersion(
+ minimalAvailableDeltaVersion + 1, latestDeltaVersion, transaction, processedDataFiles);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+
+ transaction.commitTransaction();
+
+ LOG.info(
+ "Successfully created Iceberg table {} from Delta Lake table at {}, processed data file count: {}",
+ newTableIdentifier,
+ deltaTableLocation,
+ processedDataFiles.size());
+ return processedDataFiles::size;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r3829125279
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeKernelTableAction.java:
##########
@@ -0,0 +1,609 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.delta;
+
+import io.delta.kernel.Scan;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
Review Comment:
Thank you for the pointing on it. It looks like it will required to support both catalog managed and file system base commits. I added such item to the plan https://github.com/apache/iceberg/issues/17146
The catalog managed commits supported from `Delta 4.1.0.`
Is there a way to identify if the table uses catalog managed commits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Delta: Updating Delta to Iceberg conversion [iceberg]
Posted by "vladislav-sidorovich (via GitHub)" <gi...@apache.org>.
vladislav-sidorovich commented on code in PR #15407:
URL: https://github.com/apache/iceberg/pull/15407#discussion_r4271277614
##########
dev/.rat-excludes:
##########
@@ -29,4 +29,5 @@ sitemap.xml
.python-version
**/*_index.md
**/.venv/**
+**/_delta_log/_last_checkpoint
Review Comment:
There is https://github.com/delta-incubator/dat. We can download and cache it in locally `~./m2/..` for the future builds.
I think it's the way to go:
1. No source in Iceberg repo (deps as artifact)
2. Static immutable files for Delta protocol
3. Downloaded once and cached as regular dependency.
@kevinjqliu are you ok with the plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org