You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "MehulBatra (via GitHub)" <gi...@apache.org> on 2025/07/20 15:21:13 UTC
[PR] in progress [fluss]
MehulBatra opened a new pull request, #1372:
URL: https://github.com/apache/fluss/pull/1372
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/alibaba/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
Add support for IcebergLakeCatalog for fluss primary key tables
<!-- Linking this pull request to the issue -->
Linked issue: close #1337
<!-- What is the purpose of the change -->
### Brief change log
- Added new file for IcebergLakeCatalog implementing LakeCatalog
- Modified IcebergLakeStorage.createLakeCatalog method
- In progress
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
In progress
### API and Format
<!-- Does this change affect API or storage format -->
In progress
### Documentation
<!-- Does this change introduce a new feature -->
In progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2253228561
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
Review Comment:
nit:
remove this comment.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
+
+ Schema icebergSchema = convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+ SortOrder sortOrder = createSortOrder(icebergSchema);
+
+ // table builder for complete configuration
+ Catalog.TableBuilder tableBuilder = icebergCatalog.buildTable(icebergId, icebergSchema);
+ tableBuilder.withProperties(buildTableProperties(tableDescriptor));
+ tableBuilder.withPartitionSpec(partitionSpec);
+ tableBuilder.withSortOrder(sortOrder);
+ tableBuilder.create();
+
+ } catch (AlreadyExistsException e) {
+ throw new TableAlreadyExistException(tablePath.getTableName(), e);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create Iceberg table: " + tablePath, e);
+ }
+ }
+
+ private TableIdentifier toIcebergTableIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema convertToIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ for (Map.Entry<String, Type> systemColumn : SYSTEM_COLUMNS.entrySet()) {
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, systemColumn.getKey(), systemColumn.getValue()));
+ }
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ DataTypeRoot typeRoot = dataType.getTypeRoot();
+
+ switch (typeRoot) {
+ case INTEGER:
+ return Types.IntegerType.get();
+ case BIGINT:
+ return Types.LongType.get();
+ case STRING:
+ return Types.StringType.get();
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return Types.TimestampType.withZone();
+ case TIMESTAMP_WITHOUT_TIME_ZONE:
+ return Types.TimestampType.withoutZone();
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(TableDescriptor.TableDistribution::getBucketCount)
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "Bucket count (bucket.num) must be set"));
+
+ if (bucketKeys.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Bucket key must be set for primary key Iceberg tables");
+ }
+ if (bucketKeys.size() > 1) {
+ throw new UnsupportedOperationException(
+ "Only one bucket key is supported for Iceberg at the moment");
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
+ builder.bucket(bucketKeys.get(0), bucketCount);
+ List<String> partitionKeys = tableDescriptor.getPartitionKeys();
+ for (String partitionKey : partitionKeys) {
+ builder.identity(partitionKey);
+ }
+
+ return builder.build();
+ }
+
+ private void setFlussPropertyToIceberg(
+ String key, String value, Map<String, String> icebergProperties) {
+
Review Comment:
nit: remove this blank line
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
no need always create the database as i comment before. You can refer to what paimon did in here.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
+
+ Schema icebergSchema = convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+ SortOrder sortOrder = createSortOrder(icebergSchema);
+
+ // table builder for complete configuration
+ Catalog.TableBuilder tableBuilder = icebergCatalog.buildTable(icebergId, icebergSchema);
+ tableBuilder.withProperties(buildTableProperties(tableDescriptor));
+ tableBuilder.withPartitionSpec(partitionSpec);
+ tableBuilder.withSortOrder(sortOrder);
+ tableBuilder.create();
+
+ } catch (AlreadyExistsException e) {
+ throw new TableAlreadyExistException(tablePath.getTableName(), e);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create Iceberg table: " + tablePath, e);
+ }
+ }
+
+ private TableIdentifier toIcebergTableIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema convertToIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ for (Map.Entry<String, Type> systemColumn : SYSTEM_COLUMNS.entrySet()) {
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, systemColumn.getKey(), systemColumn.getValue()));
+ }
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ DataTypeRoot typeRoot = dataType.getTypeRoot();
+
+ switch (typeRoot) {
+ case INTEGER:
+ return Types.IntegerType.get();
+ case BIGINT:
+ return Types.LongType.get();
+ case STRING:
+ return Types.StringType.get();
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return Types.TimestampType.withZone();
+ case TIMESTAMP_WITHOUT_TIME_ZONE:
+ return Types.TimestampType.withoutZone();
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(TableDescriptor.TableDistribution::getBucketCount)
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "Bucket count (bucket.num) must be set"));
+
+ if (bucketKeys.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Bucket key must be set for primary key Iceberg tables");
+ }
+ if (bucketKeys.size() > 1) {
+ throw new UnsupportedOperationException(
+ "Only one bucket key is supported for Iceberg at the moment");
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
+ builder.bucket(bucketKeys.get(0), bucketCount);
+ List<String> partitionKeys = tableDescriptor.getPartitionKeys();
+ for (String partitionKey : partitionKeys) {
Review Comment:
partition transform by fluss partition key should always come before bucket.
```
PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
List<String> partitionKeys = tableDescriptor.getPartitionKeys();
for (String partitionKey : partitionKeys) {
builder.identity(partitionKey);
}
builder.bucket(bucketKeys.get(0), bucketCount);
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
Review Comment:
remove this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2255422477
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
if you check method, I have taken care in createDatabase method (create if not exists else skip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2248311702
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
+ throw new UnsupportedOperationException("Fluss does not support UUID yet.");
+ case "TIMESTAMP_NANO":
+ throw new UnsupportedOperationException(
+ "Iceberg does not support nanosecond precision timestamps in 1.4.3");
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(td -> td.getBucketCount())
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "Bucket count (bucket.num) must be set"));
+
+ if (bucketKeys.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Bucket key must be set for primary key Iceberg tables");
+ }
+ if (bucketKeys.size() > 1) {
+ throw new UnsupportedOperationException(
+ "Only one bucket key is supported for Iceberg at the moment");
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
Review Comment:
ok adding partion spec in the table builder itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #1372:
URL: https://github.com/apache/fluss/pull/1372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2243737625
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #1372:
URL: https://github.com/apache/fluss/pull/1372#issuecomment-3124733239
@luoyuxia Please help me with the review, whenever you get time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2261104122
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
Review Comment:
Thanks for the lead @VisibleForTesting did help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #1372:
URL: https://github.com/apache/fluss/pull/1372#issuecomment-3166996891
> @MehulBatra I append a some commit to improve some code, PTAL. If you have no question, I'll merge later.
Thanks @luoyuxia for the review and feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2243740950
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
Review Comment:
thanks, will use the getTypeRoot itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2259291597
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
Please don't forget my this comment. I hope we can avoid to fire a request which is `supportsNamespaces.namespaceExists` to iceberg's catalog in every create table request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2234797010
##########
fluss-lake/fluss-lake-iceberg/pom.xml:
##########
@@ -32,13 +32,16 @@
<packaging>jar</packaging>
+ <properties>
+ <iceberg.version>1.4.3</iceberg.version>
Review Comment:
We don't need it since we already have it in parent pom
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
Review Comment:
nit: simplify to `AlreadyExistsException`. import `org.apache.iceberg.exceptions.AlreadyExistsException` in header.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
+ throw new UnsupportedOperationException("Fluss does not support UUID yet.");
+ case "TIMESTAMP_NANO":
Review Comment:
dito
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
+ throw new UnsupportedOperationException("Fluss does not support UUID yet.");
+ case "TIMESTAMP_NANO":
+ throw new UnsupportedOperationException(
+ "Iceberg does not support nanosecond precision timestamps in 1.4.3");
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(td -> td.getBucketCount())
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "Bucket count (bucket.num) must be set"));
+
+ if (bucketKeys.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Bucket key must be set for primary key Iceberg tables");
+ }
+ if (bucketKeys.size() > 1) {
+ throw new UnsupportedOperationException(
+ "Only one bucket key is supported for Iceberg at the moment");
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
Review Comment:
we need to consider Fluss partitioned table, as the FIP said, It'll be consider as Iceberg identity partition transform
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
not always call method create database. Only when database not exist exception is thrown when call `createTable`, do we need to create database.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
Review Comment:
Remove the extract code since the framework have done it before create `IcebergLakeCatalog`. See https://github.com/apache/fluss/blob/b2f9c2f6769a28fe0b0a626770208cc877d3b743/fluss-server/src/main/java/com/alibaba/fluss/server/coordinator/CoordinatorServer.java#L258
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
Review Comment:
simplify to
```
for (Map.Entry<String, Type> systemColumn : SYSTEM_COLUMNS.entrySet()) {
fields.add(
Types.NestedField.optional(
fieldId++, systemColumn.getKey(), systemColumn.getValue()));
}
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
Review Comment:
please not to use string, it's error-prone. You can use that:
```
switch (dataType.getTypeRoot()) {
case INTEGER:
return Types.IntegerType.get();
case xxx:
}
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
Review Comment:
`UUID` is not fluss type, we don't need to consider it.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
Review Comment:
nit: simplify to
`NoSuchNamespaceException`
`AlreadyExistsException`
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
+ throw new UnsupportedOperationException("Fluss does not support UUID yet.");
+ case "TIMESTAMP_NANO":
+ throw new UnsupportedOperationException(
+ "Iceberg does not support nanosecond precision timestamps in 1.4.3");
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(td -> td.getBucketCount())
Review Comment:
nit:
```suggestion
.flatMap(TableDescriptor.TableDistribution::getBucketCount)
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
Review Comment:
As the FIP said, it should sort by the system column `__offset`. So we should use that:
```
Catalog.TableBuilder tableBuilder =
icebergCatalog.buildTable(icebergId, icebergSchema);
tableBuilder.withProperties(icebergTableProperties);
tableBuilder.withPartitionSpec(partitionSpec);
tableBuilder.withSortOrder(xxxx)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2255423634
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
+
+ Schema icebergSchema = convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+ SortOrder sortOrder = createSortOrder(icebergSchema);
+
+ // table builder for complete configuration
+ Catalog.TableBuilder tableBuilder = icebergCatalog.buildTable(icebergId, icebergSchema);
+ tableBuilder.withProperties(buildTableProperties(tableDescriptor));
+ tableBuilder.withPartitionSpec(partitionSpec);
+ tableBuilder.withSortOrder(sortOrder);
+ tableBuilder.create();
+
+ } catch (AlreadyExistsException e) {
+ throw new TableAlreadyExistException(tablePath.getTableName(), e);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create Iceberg table: " + tablePath, e);
+ }
+ }
+
+ private TableIdentifier toIcebergTableIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema convertToIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ for (Map.Entry<String, Type> systemColumn : SYSTEM_COLUMNS.entrySet()) {
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, systemColumn.getKey(), systemColumn.getValue()));
+ }
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ DataTypeRoot typeRoot = dataType.getTypeRoot();
+
+ switch (typeRoot) {
+ case INTEGER:
+ return Types.IntegerType.get();
+ case BIGINT:
+ return Types.LongType.get();
+ case STRING:
+ return Types.StringType.get();
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return Types.TimestampType.withZone();
+ case TIMESTAMP_WITHOUT_TIME_ZONE:
+ return Types.TimestampType.withoutZone();
+ default:
+ throw new UnsupportedOperationException("Unsupported Fluss data type: " + typeRoot);
+ }
+ }
+
+ private PartitionSpec createPartitionSpec(
+ TableDescriptor tableDescriptor, Schema icebergSchema) {
+
+ // Only PK tables supported for now
+ List<String> bucketKeys = tableDescriptor.getBucketKeys();
+ int bucketCount =
+ tableDescriptor
+ .getTableDistribution()
+ .flatMap(TableDescriptor.TableDistribution::getBucketCount)
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ "Bucket count (bucket.num) must be set"));
+
+ if (bucketKeys.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Bucket key must be set for primary key Iceberg tables");
+ }
+ if (bucketKeys.size() > 1) {
+ throw new UnsupportedOperationException(
+ "Only one bucket key is supported for Iceberg at the moment");
+ }
+
+ PartitionSpec.Builder builder = PartitionSpec.builderFor(icebergSchema);
+ builder.bucket(bucketKeys.get(0), bucketCount);
+ List<String> partitionKeys = tableDescriptor.getPartitionKeys();
+ for (String partitionKey : partitionKeys) {
Review Comment:
oh yes missed it as per the doc we do PARTITIONED BY (bucket(order_id, 3)) my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #1372:
URL: https://github.com/apache/fluss/pull/1372#issuecomment-3170406343
@MehulBatra Thanks for the great work. I’ll follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2243738373
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
will change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2243736455
##########
fluss-lake/fluss-lake-iceberg/pom.xml:
##########
@@ -32,13 +32,16 @@
<packaging>jar</packaging>
+ <properties>
+ <iceberg.version>1.4.3</iceberg.version>
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2255665954
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
I meansactually we don't always need to call method `createDatabase` . Althogh you create if not exists else skip, but it will still try to fire a request to iceberg's catalog which is not necessary.
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.types.DataTypeRoot;
+import com.alibaba.fluss.utils.IOUtils;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.apache.iceberg.CatalogUtil.loadCatalog;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+
+ this.icebergCatalog = createIcebergCatalog(configuration);
+ }
+
+ private Catalog createIcebergCatalog(Configuration configuration) {
+ // Configuration has already been filtered by extractLakeProperties()
+ Map<String, String> icebergProps = configuration.toMap();
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ return loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ try {
+ TableIdentifier icebergId = toIcebergTableIdentifier(tablePath);
+
+ // Ensure namespace exists
+ createDatabase(tablePath.getDatabaseName());
Review Comment:
I means actually we don't always need to call method `createDatabase` . Althogh you create if not exists else skip, but it will still try to fire a request to iceberg's catalog which is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2234762809
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
Review Comment:
The import for LakeCatalog is missing. This will cause a compilation error since the class implements LakeCatalog interface but doesn't import it.
```suggestion
import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
Review Comment:
The fieldId is being reused for multiple system columns. Each field should have a unique ID. The second and third system columns should use fieldId+1 and fieldId+2 for their IDs respectively.
```suggestion
Types.NestedField.optional(fieldId++, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
fields.add(
Types.NestedField.optional(fieldId++, OFFSET_COLUMN_NAME, Types.LongType.get()));
fields.add(
Types.NestedField.optional(
fieldId++, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
+
+ return new Schema(fields);
+ }
+
+ private Type convertFlussToIcebergType(DataType dataType) {
+ String typeRoot = dataType.getTypeRoot().name();
+
+ switch (typeRoot) {
+ case "INT":
+ return Types.IntegerType.get();
+ case "BIGINT":
+ return Types.LongType.get();
+ case "STRING":
+ return Types.StringType.get();
+ case "BOOLEAN":
+ return Types.BooleanType.get();
+ case "DOUBLE":
+ return Types.DoubleType.get();
+ case "FLOAT":
+ return Types.FloatType.get();
+ case "DATE":
+ return Types.DateType.get();
+ case "TIMESTAMP_WITH_LOCAL_TIME_ZONE":
+ return Types.TimestampType.withZone();
+ case "TIMESTAMP_WITHOUT_TIME_ZONE":
+ return Types.TimestampType.withoutZone();
+ case "UUID":
+ throw new UnsupportedOperationException("Fluss does not support UUID yet.");
+ case "TIMESTAMP_NANO":
+ throw new UnsupportedOperationException(
+ "Iceberg does not support nanosecond precision timestamps in 1.4.3");
Review Comment:
The hardcoded version '1.4.3' in the error message should be replaced with a reference to the actual Iceberg version being used to maintain consistency when the version changes.
```suggestion
"Iceberg does not support nanosecond precision timestamps in " + org.apache.iceberg.Version.getVersion());
```
##########
fluss-lake/fluss-lake-iceberg/src/main/java/com/alibaba/fluss/lake/iceberg/IcebergLakeCatalog.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.lake.iceberg;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.exception.TableAlreadyExistException;
+import com.alibaba.fluss.lake.lakestorage.LakeCatalog;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataType;
+import com.alibaba.fluss.utils.IOUtils;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+
+/** A Iceberg implementation of {@link LakeCatalog}. */
+public class IcebergLakeCatalog implements LakeCatalog {
+
+ private static final LinkedHashMap<String, Type> SYSTEM_COLUMNS = new LinkedHashMap<>();
+
+ static {
+ // We need __bucket system column to filter out the given bucket
+ // for iceberg bucket append only table & primary key table.
+ SYSTEM_COLUMNS.put(BUCKET_COLUMN_NAME, Types.IntegerType.get());
+ SYSTEM_COLUMNS.put(OFFSET_COLUMN_NAME, Types.LongType.get());
+ SYSTEM_COLUMNS.put(TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone());
+ }
+
+ private final Catalog icebergCatalog;
+
+ // for fluss config
+ private static final String FLUSS_CONF_PREFIX = "fluss.";
+ // for iceberg config
+ private static final String ICEBERG_CONF_PREFIX = "iceberg.";
+ private static final String ICEBERG_CATALOG_PREFIX = "iceberg.catalog.";
+
+ public IcebergLakeCatalog(Configuration configuration) {
+ // Extract Iceberg catalog properties from Fluss configuration
+ Map<String, String> icebergProps =
+ configuration.toMap().entrySet().stream()
+ .filter(e -> e.getKey().startsWith(ICEBERG_CATALOG_PREFIX))
+ .collect(
+ Collectors.toMap(
+ e -> e.getKey().substring(ICEBERG_CATALOG_PREFIX.length()),
+ Map.Entry::getValue));
+
+ String catalogType = icebergProps.get("type");
+ if (catalogType == null) {
+ throw new IllegalArgumentException(
+ "Missing required Iceberg catalog type. Set 'iceberg.catalog.type' in your configuration (e.g., 'hive', 'hadoop', or 'rest').");
+ }
+ String catalogName = icebergProps.getOrDefault("name", "fluss-iceberg-catalog");
+
+ this.icebergCatalog =
+ org.apache.iceberg.CatalogUtil.loadCatalog(
+ catalogType,
+ catalogName,
+ icebergProps,
+ null // Optional: pass Hadoop configuration if available
+ );
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, TableDescriptor tableDescriptor)
+ throws TableAlreadyExistException {
+ if (!tableDescriptor.hasPrimaryKey()) {
+ throw new UnsupportedOperationException(
+ "Only primary key tables are supported currently.");
+ }
+
+ TableIdentifier icebergId = toIcebergIdentifier(tablePath);
+ Schema icebergSchema = toIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec = createPartitionSpec(tableDescriptor, icebergSchema);
+
+ Map<String, String> icebergProperties = new HashMap<>();
+
+ // MOR table properties
+ icebergProperties.put("write.delete.mode", "merge-on-read");
+ icebergProperties.put("write.update.mode", "merge-on-read");
+ icebergProperties.put("write.merge.mode", "merge-on-read");
+
+ tableDescriptor
+ .getProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+ tableDescriptor
+ .getCustomProperties()
+ .forEach((k, v) -> setFlussPropertyToIceberg(k, v, icebergProperties));
+
+ createDatabase(tablePath.getDatabaseName());
+
+ try {
+ createTable(icebergId, icebergSchema, partitionSpec, icebergProperties);
+ } catch (org.apache.iceberg.exceptions.AlreadyExistsException e) {
+ throw new TableAlreadyExistException("Table " + tablePath + " already exists.");
+ }
+ }
+
+ private void createTable(
+ TableIdentifier icebergId,
+ Schema icebergSchema,
+ PartitionSpec partitionSpec,
+ Map<String, String> icebergTableProperties)
+ throws org.apache.iceberg.exceptions.NoSuchNamespaceException,
+ org.apache.iceberg.exceptions.AlreadyExistsException {
+
+ icebergCatalog.createTable(icebergId, icebergSchema, partitionSpec, icebergTableProperties);
+ }
+
+ private TableIdentifier toIcebergIdentifier(TablePath tablePath) {
+ return TableIdentifier.of(tablePath.getDatabaseName(), tablePath.getTableName());
+ }
+
+ private Schema toIcebergSchema(TableDescriptor tableDescriptor) {
+ List<Types.NestedField> fields = new ArrayList<>();
+ int fieldId = 1;
+
+ for (com.alibaba.fluss.metadata.Schema.Column column :
+ tableDescriptor.getSchema().getColumns()) {
+ String colName = column.getName();
+ if (SYSTEM_COLUMNS.containsKey(colName)) {
+ throw new IllegalArgumentException(
+ "Column '" + colName + "' conflicts with a reserved system column name.");
+ }
+ fields.add(
+ Types.NestedField.optional(
+ fieldId++, colName, convertFlussToIcebergType(column.getDataType())));
+ }
+
+ fields.add(
+ Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
+ fields.add(
+ Types.NestedField.optional(fieldId + 1, OFFSET_COLUMN_NAME, Types.LongType.get()));
+ fields.add(
+ Types.NestedField.optional(
+ fieldId + 2, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
Review Comment:
This line uses the same fieldId as the previous field. It should use fieldId++ to increment the ID before using it.
```suggestion
fieldId++;
fields.add(
Types.NestedField.optional(fieldId, BUCKET_COLUMN_NAME, Types.IntegerType.get()));
fieldId++;
fields.add(
Types.NestedField.optional(fieldId, OFFSET_COLUMN_NAME, Types.LongType.get()));
fieldId++;
fields.add(
Types.NestedField.optional(fieldId, TIMESTAMP_COLUMN_NAME, Types.TimestampType.withZone()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2261110950
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
+
+ assertThat(createdTable).isNotNull();
+ assertThat(createdTable.name()).endsWith(tableName);
+ assertThat(createdTable.schema().columns())
+ .hasSize(2)
+ .extracting("name")
+ .containsExactly("id", "name");
+ }
+
+ @Test
+ void convertsFlussSchemaToIcebergWithSystemColumns() {
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("order_id", DataTypes.INT())
+ .column("customer_name", DataTypes.STRING())
+ .column("created_at", DataTypes.TIMESTAMP_LTZ()) // timestamp with zone
+ .column("processed_at", DataTypes.TIMESTAMP()) // timestamp without zone
+ .primaryKey("order_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "order_id").build();
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ assertThat(icebergSchema.findField("order_id")).isNotNull();
+ assertThat(icebergSchema.findField("customer_name")).isNotNull();
+ assertThat(icebergSchema.findField("created_at")).isNotNull();
+ assertThat(icebergSchema.findField("processed_at")).isNotNull();
+
+ assertThat(icebergSchema.findField("created_at").type())
+ .isEqualTo(Types.TimestampType.withZone());
+ assertThat(icebergSchema.findField("processed_at").type())
+ .isEqualTo(Types.TimestampType.withoutZone());
+
+ assertThat(icebergSchema.findField(BUCKET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(OFFSET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(TIMESTAMP_COLUMN_NAME)).isNotNull();
+
+ assertThat(icebergSchema.columns()).hasSize(7);
+ }
+
+ @Test
+ void createsPrimaryKeyTableBucketPartitionedOnly() {
+ String database = "test_db";
+ String tableName = "pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ // primary key table
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("shop_id", DataTypes.BIGINT())
+ .column("user_id", DataTypes.BIGINT())
+ .column("num_orders", DataTypes.INT())
+ .column("total_amount", DataTypes.INT())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ PartitionSpec expectedPartitionSpec =
+ flussIcebergCatalog.createPartitionSpec(tableDescriptor, icebergSchema);
+
+ SortOrder expectedSortOrder =
+ SortOrder.builderFor(icebergSchema).asc(OFFSET_COLUMN_NAME).build();
+
+ Map<String, String> tableProperties = new HashMap<>();
+ tableProperties.put("write.merge.mode", "merge-on-read");
+
+ Table createdTable =
+ icebergCatalog
+ .buildTable(tableId, icebergSchema)
+ .withPartitionSpec(expectedPartitionSpec)
+ .withSortOrder(expectedSortOrder)
+ .withProperties(tableProperties)
+ .create();
+
+ assertThat(createdTable.spec().fields()).hasSize(1);
+ assertThat(createdTable.spec().fields().get(0).name()).isEqualTo("user_id_bucket");
+ assertThat(createdTable.spec().fields().get(0).transform().toString())
+ .contains("bucket[4]");
+
+ assertThat(createdTable.sortOrder().fields()).hasSize(1);
+ assertThat(createdTable.sortOrder().fields().get(0).sourceId())
+ .isEqualTo(icebergSchema.findField(OFFSET_COLUMN_NAME).fieldId());
+
+ assertThat(createdTable.properties()).containsEntry("write.merge.mode", "merge-on-read");
+ }
+
+ @Test
+ void rejectsPrimaryKeyTableWithMultipleBucketKeys() {
+ String database = "test_db";
+ String tableName = "multi_bucket_pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("shop_id", DataTypes.BIGINT())
+ .column("order_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id", "shop_id") // Composite primary key
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder()
+ .schema(flussSchema)
+ .distributedBy(4, "user_id", "shop_id") // Multiple bucket keys
+ .build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ assertThatThrownBy(
+ () -> {
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec =
+ flussIcebergCatalog.createPartitionSpec(
+ tableDescriptor, icebergSchema);
+ icebergCatalog.createTable(tableId, icebergSchema, partitionSpec);
+ })
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Only one bucket key is supported for Iceberg");
+ }
+
+ @Test
+ void verifiesFlussCreatesValidIcebergTable() throws Exception {
+ String database = "test_db";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ // Create via Fluss iceberg catalog
+ TablePath flussTablePath = TablePath.of(database, "fluss_created");
+ flussIcebergCatalog.createTable(flussTablePath, tableDescriptor);
Review Comment:
My bad I got misaligned by referring to PaimonTieringTest createTable method, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2259317853
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
Review Comment:
We can remove this method. Actaully, we don't care about iceberg's catalog can create table or not since it's iceberg implementation. We need care about our `IcebergLakeCatalog` should create table successfully.
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
Review Comment:
can we use the constructor `IcebergLakeCatalog(Configuration configuration)` to verify we can create from configuration?
You can add a `@VisibleForTesting` in `IcebergLakeCatalog` to return the created iceberg catalog.
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
+
+ assertThat(createdTable).isNotNull();
+ assertThat(createdTable.name()).endsWith(tableName);
+ assertThat(createdTable.schema().columns())
+ .hasSize(2)
+ .extracting("name")
+ .containsExactly("id", "name");
+ }
+
+ @Test
+ void convertsFlussSchemaToIcebergWithSystemColumns() {
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("order_id", DataTypes.INT())
+ .column("customer_name", DataTypes.STRING())
+ .column("created_at", DataTypes.TIMESTAMP_LTZ()) // timestamp with zone
+ .column("processed_at", DataTypes.TIMESTAMP()) // timestamp without zone
+ .primaryKey("order_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "order_id").build();
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ assertThat(icebergSchema.findField("order_id")).isNotNull();
+ assertThat(icebergSchema.findField("customer_name")).isNotNull();
+ assertThat(icebergSchema.findField("created_at")).isNotNull();
+ assertThat(icebergSchema.findField("processed_at")).isNotNull();
+
+ assertThat(icebergSchema.findField("created_at").type())
+ .isEqualTo(Types.TimestampType.withZone());
+ assertThat(icebergSchema.findField("processed_at").type())
+ .isEqualTo(Types.TimestampType.withoutZone());
+
+ assertThat(icebergSchema.findField(BUCKET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(OFFSET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(TIMESTAMP_COLUMN_NAME)).isNotNull();
+
+ assertThat(icebergSchema.columns()).hasSize(7);
+ }
+
+ @Test
+ void createsPrimaryKeyTableBucketPartitionedOnly() {
+ String database = "test_db";
+ String tableName = "pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ // primary key table
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("shop_id", DataTypes.BIGINT())
+ .column("user_id", DataTypes.BIGINT())
+ .column("num_orders", DataTypes.INT())
+ .column("total_amount", DataTypes.INT())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ PartitionSpec expectedPartitionSpec =
+ flussIcebergCatalog.createPartitionSpec(tableDescriptor, icebergSchema);
+
+ SortOrder expectedSortOrder =
+ SortOrder.builderFor(icebergSchema).asc(OFFSET_COLUMN_NAME).build();
+
+ Map<String, String> tableProperties = new HashMap<>();
+ tableProperties.put("write.merge.mode", "merge-on-read");
+
+ Table createdTable =
+ icebergCatalog
+ .buildTable(tableId, icebergSchema)
+ .withPartitionSpec(expectedPartitionSpec)
+ .withSortOrder(expectedSortOrder)
+ .withProperties(tableProperties)
+ .create();
+
+ assertThat(createdTable.spec().fields()).hasSize(1);
+ assertThat(createdTable.spec().fields().get(0).name()).isEqualTo("user_id_bucket");
+ assertThat(createdTable.spec().fields().get(0).transform().toString())
+ .contains("bucket[4]");
+
+ assertThat(createdTable.sortOrder().fields()).hasSize(1);
+ assertThat(createdTable.sortOrder().fields().get(0).sourceId())
+ .isEqualTo(icebergSchema.findField(OFFSET_COLUMN_NAME).fieldId());
+
+ assertThat(createdTable.properties()).containsEntry("write.merge.mode", "merge-on-read");
+ }
+
+ @Test
+ void rejectsPrimaryKeyTableWithMultipleBucketKeys() {
+ String database = "test_db";
+ String tableName = "multi_bucket_pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("shop_id", DataTypes.BIGINT())
+ .column("order_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id", "shop_id") // Composite primary key
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder()
+ .schema(flussSchema)
+ .distributedBy(4, "user_id", "shop_id") // Multiple bucket keys
+ .build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ assertThatThrownBy(
+ () -> {
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec =
+ flussIcebergCatalog.createPartitionSpec(
+ tableDescriptor, icebergSchema);
+ icebergCatalog.createTable(tableId, icebergSchema, partitionSpec);
+ })
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Only one bucket key is supported for Iceberg");
+ }
+
+ @Test
+ void verifiesFlussCreatesValidIcebergTable() throws Exception {
+ String database = "test_db";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ // Create via Fluss iceberg catalog
+ TablePath flussTablePath = TablePath.of(database, "fluss_created");
+ flussIcebergCatalog.createTable(flussTablePath, tableDescriptor);
+
+ // Create equivalent via raw Iceberg catalog
+ org.apache.iceberg.Schema pureIcebergSchema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.optional(1, "user_id", Types.LongType.get()),
+ Types.NestedField.optional(2, "amount", Types.DoubleType.get()));
+
+ TableIdentifier pureTableId = TableIdentifier.of(database, "pure_iceberg");
+ Table pureIcebergTable = icebergCatalog.createTable(pureTableId, pureIcebergSchema);
Review Comment:
we don't care about iceberg's catalog can create table or not since it's iceberg implementation.
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
+
+ assertThat(createdTable).isNotNull();
+ assertThat(createdTable.name()).endsWith(tableName);
+ assertThat(createdTable.schema().columns())
+ .hasSize(2)
+ .extracting("name")
+ .containsExactly("id", "name");
+ }
+
+ @Test
+ void convertsFlussSchemaToIcebergWithSystemColumns() {
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("order_id", DataTypes.INT())
+ .column("customer_name", DataTypes.STRING())
+ .column("created_at", DataTypes.TIMESTAMP_LTZ()) // timestamp with zone
+ .column("processed_at", DataTypes.TIMESTAMP()) // timestamp without zone
+ .primaryKey("order_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "order_id").build();
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ assertThat(icebergSchema.findField("order_id")).isNotNull();
+ assertThat(icebergSchema.findField("customer_name")).isNotNull();
+ assertThat(icebergSchema.findField("created_at")).isNotNull();
+ assertThat(icebergSchema.findField("processed_at")).isNotNull();
+
+ assertThat(icebergSchema.findField("created_at").type())
+ .isEqualTo(Types.TimestampType.withZone());
+ assertThat(icebergSchema.findField("processed_at").type())
+ .isEqualTo(Types.TimestampType.withoutZone());
+
+ assertThat(icebergSchema.findField(BUCKET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(OFFSET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(TIMESTAMP_COLUMN_NAME)).isNotNull();
+
+ assertThat(icebergSchema.columns()).hasSize(7);
+ }
+
+ @Test
+ void createsPrimaryKeyTableBucketPartitionedOnly() {
+ String database = "test_db";
+ String tableName = "pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ // primary key table
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("shop_id", DataTypes.BIGINT())
+ .column("user_id", DataTypes.BIGINT())
+ .column("num_orders", DataTypes.INT())
+ .column("total_amount", DataTypes.INT())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ PartitionSpec expectedPartitionSpec =
+ flussIcebergCatalog.createPartitionSpec(tableDescriptor, icebergSchema);
+
+ SortOrder expectedSortOrder =
+ SortOrder.builderFor(icebergSchema).asc(OFFSET_COLUMN_NAME).build();
+
+ Map<String, String> tableProperties = new HashMap<>();
+ tableProperties.put("write.merge.mode", "merge-on-read");
+
+ Table createdTable =
+ icebergCatalog
+ .buildTable(tableId, icebergSchema)
+ .withPartitionSpec(expectedPartitionSpec)
+ .withSortOrder(expectedSortOrder)
+ .withProperties(tableProperties)
+ .create();
+
+ assertThat(createdTable.spec().fields()).hasSize(1);
+ assertThat(createdTable.spec().fields().get(0).name()).isEqualTo("user_id_bucket");
+ assertThat(createdTable.spec().fields().get(0).transform().toString())
+ .contains("bucket[4]");
+
+ assertThat(createdTable.sortOrder().fields()).hasSize(1);
+ assertThat(createdTable.sortOrder().fields().get(0).sourceId())
+ .isEqualTo(icebergSchema.findField(OFFSET_COLUMN_NAME).fieldId());
+
+ assertThat(createdTable.properties()).containsEntry("write.merge.mode", "merge-on-read");
+ }
+
+ @Test
+ void rejectsPrimaryKeyTableWithMultipleBucketKeys() {
+ String database = "test_db";
+ String tableName = "multi_bucket_pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("shop_id", DataTypes.BIGINT())
+ .column("order_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id", "shop_id") // Composite primary key
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder()
+ .schema(flussSchema)
+ .distributedBy(4, "user_id", "shop_id") // Multiple bucket keys
+ .build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ assertThatThrownBy(
+ () -> {
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec =
+ flussIcebergCatalog.createPartitionSpec(
+ tableDescriptor, icebergSchema);
+ icebergCatalog.createTable(tableId, icebergSchema, partitionSpec);
+ })
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Only one bucket key is supported for Iceberg");
+ }
+
+ @Test
+ void verifiesFlussCreatesValidIcebergTable() throws Exception {
+ String database = "test_db";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ // Create via Fluss iceberg catalog
+ TablePath flussTablePath = TablePath.of(database, "fluss_created");
+ flussIcebergCatalog.createTable(flussTablePath, tableDescriptor);
Review Comment:
That's what we need in this test, we need to use Fluss `IcebergLakeCatalog` to create table.
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
Review Comment:
nit: we don't need to call method `createNamespaceIfAbsent` since `IcebergLakeCatalog` should create it if it not exist.
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
+
+ assertThat(createdTable).isNotNull();
+ assertThat(createdTable.name()).endsWith(tableName);
+ assertThat(createdTable.schema().columns())
+ .hasSize(2)
+ .extracting("name")
+ .containsExactly("id", "name");
+ }
+
+ @Test
+ void convertsFlussSchemaToIcebergWithSystemColumns() {
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("order_id", DataTypes.INT())
+ .column("customer_name", DataTypes.STRING())
+ .column("created_at", DataTypes.TIMESTAMP_LTZ()) // timestamp with zone
+ .column("processed_at", DataTypes.TIMESTAMP()) // timestamp without zone
+ .primaryKey("order_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "order_id").build();
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ assertThat(icebergSchema.findField("order_id")).isNotNull();
+ assertThat(icebergSchema.findField("customer_name")).isNotNull();
+ assertThat(icebergSchema.findField("created_at")).isNotNull();
+ assertThat(icebergSchema.findField("processed_at")).isNotNull();
+
+ assertThat(icebergSchema.findField("created_at").type())
+ .isEqualTo(Types.TimestampType.withZone());
+ assertThat(icebergSchema.findField("processed_at").type())
+ .isEqualTo(Types.TimestampType.withoutZone());
+
+ assertThat(icebergSchema.findField(BUCKET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(OFFSET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(TIMESTAMP_COLUMN_NAME)).isNotNull();
+
+ assertThat(icebergSchema.columns()).hasSize(7);
+ }
+
+ @Test
+ void createsPrimaryKeyTableBucketPartitionedOnly() {
+ String database = "test_db";
+ String tableName = "pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ // primary key table
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("shop_id", DataTypes.BIGINT())
+ .column("user_id", DataTypes.BIGINT())
+ .column("num_orders", DataTypes.INT())
+ .column("total_amount", DataTypes.INT())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ PartitionSpec expectedPartitionSpec =
+ flussIcebergCatalog.createPartitionSpec(tableDescriptor, icebergSchema);
+
+ SortOrder expectedSortOrder =
+ SortOrder.builderFor(icebergSchema).asc(OFFSET_COLUMN_NAME).build();
+
+ Map<String, String> tableProperties = new HashMap<>();
+ tableProperties.put("write.merge.mode", "merge-on-read");
+
+ Table createdTable =
+ icebergCatalog
Review Comment:
we don't care about iceberg's catalog can create table or not since it's iceberg implementation. So, no need to verify the table create via Iceberg catalog provided by iceberg. We need to verify the table create via Fluss `IcebergLakeCatalog`
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+
+ Table createdTable = icebergCatalog.createTable(tableId, schema);
+
+ assertThat(createdTable).isNotNull();
+ assertThat(createdTable.name()).endsWith(tableName);
+ assertThat(createdTable.schema().columns())
+ .hasSize(2)
+ .extracting("name")
+ .containsExactly("id", "name");
+ }
+
+ @Test
+ void convertsFlussSchemaToIcebergWithSystemColumns() {
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("order_id", DataTypes.INT())
+ .column("customer_name", DataTypes.STRING())
+ .column("created_at", DataTypes.TIMESTAMP_LTZ()) // timestamp with zone
+ .column("processed_at", DataTypes.TIMESTAMP()) // timestamp without zone
+ .primaryKey("order_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "order_id").build();
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ assertThat(icebergSchema.findField("order_id")).isNotNull();
+ assertThat(icebergSchema.findField("customer_name")).isNotNull();
+ assertThat(icebergSchema.findField("created_at")).isNotNull();
+ assertThat(icebergSchema.findField("processed_at")).isNotNull();
+
+ assertThat(icebergSchema.findField("created_at").type())
+ .isEqualTo(Types.TimestampType.withZone());
+ assertThat(icebergSchema.findField("processed_at").type())
+ .isEqualTo(Types.TimestampType.withoutZone());
+
+ assertThat(icebergSchema.findField(BUCKET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(OFFSET_COLUMN_NAME)).isNotNull();
+ assertThat(icebergSchema.findField(TIMESTAMP_COLUMN_NAME)).isNotNull();
+
+ assertThat(icebergSchema.columns()).hasSize(7);
+ }
+
+ @Test
+ void createsPrimaryKeyTableBucketPartitionedOnly() {
+ String database = "test_db";
+ String tableName = "pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ // primary key table
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("shop_id", DataTypes.BIGINT())
+ .column("user_id", DataTypes.BIGINT())
+ .column("num_orders", DataTypes.INT())
+ .column("total_amount", DataTypes.INT())
+ .primaryKey("user_id")
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder().schema(flussSchema).distributedBy(4, "user_id").build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+
+ PartitionSpec expectedPartitionSpec =
+ flussIcebergCatalog.createPartitionSpec(tableDescriptor, icebergSchema);
+
+ SortOrder expectedSortOrder =
+ SortOrder.builderFor(icebergSchema).asc(OFFSET_COLUMN_NAME).build();
+
+ Map<String, String> tableProperties = new HashMap<>();
+ tableProperties.put("write.merge.mode", "merge-on-read");
+
+ Table createdTable =
+ icebergCatalog
+ .buildTable(tableId, icebergSchema)
+ .withPartitionSpec(expectedPartitionSpec)
+ .withSortOrder(expectedSortOrder)
+ .withProperties(tableProperties)
+ .create();
+
+ assertThat(createdTable.spec().fields()).hasSize(1);
+ assertThat(createdTable.spec().fields().get(0).name()).isEqualTo("user_id_bucket");
+ assertThat(createdTable.spec().fields().get(0).transform().toString())
+ .contains("bucket[4]");
+
+ assertThat(createdTable.sortOrder().fields()).hasSize(1);
+ assertThat(createdTable.sortOrder().fields().get(0).sourceId())
+ .isEqualTo(icebergSchema.findField(OFFSET_COLUMN_NAME).fieldId());
+
+ assertThat(createdTable.properties()).containsEntry("write.merge.mode", "merge-on-read");
+ }
+
+ @Test
+ void rejectsPrimaryKeyTableWithMultipleBucketKeys() {
+ String database = "test_db";
+ String tableName = "multi_bucket_pk_table";
+
+ createNamespaceIfAbsent(database);
+
+ Schema flussSchema =
+ Schema.newBuilder()
+ .column("user_id", DataTypes.BIGINT())
+ .column("shop_id", DataTypes.BIGINT())
+ .column("order_id", DataTypes.BIGINT())
+ .column("amount", DataTypes.DOUBLE())
+ .primaryKey("user_id", "shop_id") // Composite primary key
+ .build();
+
+ TableDescriptor tableDescriptor =
+ TableDescriptor.builder()
+ .schema(flussSchema)
+ .distributedBy(4, "user_id", "shop_id") // Multiple bucket keys
+ .build();
+
+ TableIdentifier tableId = TableIdentifier.of(database, tableName);
+
+ assertThatThrownBy(
+ () -> {
+ org.apache.iceberg.Schema icebergSchema =
+ flussIcebergCatalog.convertToIcebergSchema(tableDescriptor);
+ PartitionSpec partitionSpec =
+ flussIcebergCatalog.createPartitionSpec(
+ tableDescriptor, icebergSchema);
+ icebergCatalog.createTable(tableId, icebergSchema, partitionSpec);
Review Comment:
dito: we don't care about iceberg's catalog can create table or not since it's iceberg implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1372:
URL: https://github.com/apache/fluss/pull/1372#discussion_r2261102530
##########
fluss-lake/fluss-lake-iceberg/src/test/java/com/apache/fluss/lake/iceberg/catalog/IcebergLakeCatalogTest.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apache.fluss.lake.iceberg.catalog;
+
+import com.alibaba.fluss.config.Configuration;
+import com.alibaba.fluss.lake.iceberg.IcebergLakeCatalog;
+import com.alibaba.fluss.metadata.Schema;
+import com.alibaba.fluss.metadata.TableDescriptor;
+import com.alibaba.fluss.metadata.TablePath;
+import com.alibaba.fluss.types.DataTypes;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.alibaba.fluss.metadata.TableDescriptor.BUCKET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.OFFSET_COLUMN_NAME;
+import static com.alibaba.fluss.metadata.TableDescriptor.TIMESTAMP_COLUMN_NAME;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Unit test for {@link IcebergLakeCatalog}. */
+class IcebergLakeCatalogTest {
+
+ @TempDir private File tempWarehouseDir;
+
+ private Configuration configuration;
+ private Catalog icebergCatalog;
+ private SupportsNamespaces namespaceSupport;
+ private IcebergLakeCatalog flussIcebergCatalog;
+
+ @BeforeEach
+ void setupCatalog() {
+ configuration = new Configuration();
+ configuration.setString("warehouse", tempWarehouseDir.toURI().toString());
+
+ InMemoryCatalog inMemoryCatalog = new InMemoryCatalog();
+ inMemoryCatalog.initialize(
+ "test_catalog", ImmutableMap.of("warehouse", tempWarehouseDir.toURI().toString()));
+
+ this.icebergCatalog = inMemoryCatalog;
+ this.namespaceSupport = inMemoryCatalog;
+ this.flussIcebergCatalog = new IcebergLakeCatalog(inMemoryCatalog);
+ }
+
+ @Test
+ void verifiesBasicTableIdentifierAndWarehouseConfig() {
+ TableIdentifier tableId = TableIdentifier.of("test_db", "simple_table");
+
+ assertThat(tableId.name()).isEqualTo("simple_table");
+ assertThat(tableId.namespace().toString()).isEqualTo("test_db");
+
+ assertThat(configuration.toMap().get("warehouse"))
+ .isEqualTo(tempWarehouseDir.toURI().toString());
+ assertThat(icebergCatalog).isNotNull();
+ }
+
+ @Test
+ void createsTableInNewNamespaceSuccessfully() {
+ String database = "test_db";
+ String tableName = "simple_table";
+
+ createNamespaceIfAbsent(database);
Review Comment:
yes with flussIcebergCatalog it should work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/iceberg] Implement IcebergLakeCatalog for PK Tables [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #1372:
URL: https://github.com/apache/fluss/pull/1372#issuecomment-3166731642
@MehulBatra I append a some commit to improve some code, PTAL. If you have no question, I'll merge later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org