[GitHub] [incubator-doris] qidaye commented on a diff in pull request #9658: Refactor part of Iceberg code to support more Iceberg Catalogs

[GitBox <gi...@apache.org>]

qidaye commented on code in PR #9658:
URL: https://github.com/apache/incubator-doris/pull/9658#discussion_r875717698


##########
fe/fe-core/src/main/java/org/apache/doris/external/iceberg/IcebergCatalogMgr.java:
##########
@@ -50,83 +49,62 @@ public class IcebergCatalogMgr {
 
     // hive metastore uri -> iceberg catalog
     // used to cache iceberg catalogs
-    private static final ConcurrentHashMap<String, IcebergCatalog> metastoreUriToCatalog = new ConcurrentHashMap();
+    private static final ConcurrentHashMap<IcebergProperty, IcebergCatalog> propertiesToCatalog = new ConcurrentHashMap();

Review Comment:
   It's better to use `IcebergProperty.extraProperties`, such as `uri`/`warehouse`, as the cache key. 
   `IcebergProperty` is bound to the database and table, not a generic cache.



##########
fe/fe-core/src/main/java/org/apache/doris/planner/IcebergScanNode.java:
##########
@@ -99,9 +98,6 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) {
         StringBuilder output = new StringBuilder();
         if (!isLoad()) {
             output.append(prefix).append("TABLE: ").append(icebergTable.getName()).append("\n");
-            output.append(prefix).append("PATH: ")

Review Comment:
   Uri or warehouse information should be added here.



##########
fe/fe-core/src/main/java/org/apache/doris/external/iceberg/IcebergCatalogImpl.java:
##########
@@ -14,54 +14,48 @@
 // KIND, either express or implied.  See the License for the
 // specific language governing permissions and limitations
 // under the License.
-
 package org.apache.doris.external.iceberg;
 
+import com.google.common.collect.Maps;
 import org.apache.doris.catalog.IcebergProperty;
-
 import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogUtil;
 import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
 import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.SupportsNamespaces;
 import org.apache.iceberg.catalog.TableIdentifier;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
-/**
- * HiveCatalog of Iceberg
- */
-public class HiveCatalog implements IcebergCatalog {
-    private static final Logger LOG = LogManager.getLogger(HiveCatalog.class);
+public class IcebergCatalogImpl implements IcebergCatalog {
+    private static final Logger LOG = LogManager.getLogger(IcebergCatalogImpl.class);
 
-    private org.apache.iceberg.hive.HiveCatalog hiveCatalog;
-
-    public HiveCatalog() {
-        hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
-    }
+    private String catalogType;
+    private Catalog icebergCatalog;
 
     @Override
     public void initialize(IcebergProperty icebergProperty) {
-        // set hadoop conf
+        this.catalogType = icebergProperty.getCatalogType();
+        Map<String, String> properties = Maps.newHashMap(icebergProperty.getExtraProperties());
+        properties.put(CatalogUtil.ICEBERG_CATALOG_TYPE, icebergProperty.getCatalogType());
+        // Hadoop configuration
         Configuration conf = new Configuration();
-        hiveCatalog.setConf(conf);
-        // initialize hive catalog
-        Map<String, String> catalogProperties = new HashMap<>();
-        catalogProperties.put("uri", icebergProperty.getHiveMetastoreUris());
-        hiveCatalog.initialize("hive", catalogProperties);
+        this.icebergCatalog = CatalogUtil.buildIcebergCatalog(icebergProperty.getCatalogType(), properties, conf);

Review Comment:
   **Note**: `CatalogUtil.buildIcebergCatalog` only supports `HiveCatalog` and `HadoopCatalog` for now.
   How do you plan to support other catalog types?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org