You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "kroushan-nit (via GitHub)" <gi...@apache.org> on 2024/12/12 17:21:55 UTC

[PR] 590 iceberg glue sync [incubator-xtable]

kroushan-nit opened a new pull request, #599:
URL: https://github.com/apache/incubator-xtable/pull/599

   ## *Important Read*
   - *Please ensure the GitHub issue is mentioned at the beginning of the PR*
   
   ## What is the purpose of the pull request
   
   *(For example: This pull request implements the sync for delta format.)*
   
   ## Brief change log
   
   *(for example:)*
   - *Fixed JSON parsing error when persisting state*
   - *Added unit tests for schema evolution*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
   - *Added integration tests for end-to-end.*
   - *Added TestConversionController to verify the change.*
   - *Manually verified the change by running a job locally.*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1882701348


##########
xtable-core/src/main/java/org/apache/xtable/exception/CatalogSyncException.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.exception;
+
+import org.apache.xtable.model.exception.ErrorCode;
+import org.apache.xtable.model.exception.InternalException;
+
+public class CatalogSyncException extends InternalException {

Review Comment:
   TODO: Add java doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900151583


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(

Review Comment:
   I am worried that this is very Iceberg specific and will also limit our extensibility with new formats. Since this will throw an exception if a new format is detected, the user cannot plugin in a new implementation for a  new format.
   



##########
xtable-utilities/src/main/resources/xtable-hadoop-defaults.xml:
##########
@@ -55,15 +55,15 @@
   </property>
   <property>
     <name>fs.s3.aws.credentials.provider</name>
-    <value>com.amazonaws.auth.DefaultAWSCredentialsProviderChain</value>
+    <value>software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider</value>

Review Comment:
   It could be useful to pull the aws and hadoop upgrade related code into a smaller PR that can be merged quickly for anyone that may be running into the compatibility issues you saw with hadoop and aws sdk



##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Use proper javadocs here please. 
   
   Also will all catalogs be required to set this property?



##########
xtable-core/src/main/java/org/apache/xtable/catalog/Constants.java:
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+public class Constants {

Review Comment:
   Similar nitpick here, use NoArgsConstructor with private access



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1886702521


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+public class TableFormatUtils {
+
+  /** Get iceberg table data files location */
+  public static String getIcebergDataLocation(

Review Comment:
   @vinishjail97 This will be reused across catalog sync clients, so created a utility class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900241845


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Does it make more sense to move that logic into the method instead of making each caller throw that exception?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900183692


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Why will the spark type be set? There is no guarantee spark created the table. 
   What should the code do if tableFormat is still null at the end of the method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900195614


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   - As per the documentation: https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables-syncing-metadata.html, Athena and AWS Glue consoles requires `spark.sql.sources.provider` property to be set when creating delta lake table using those methods.
   
   - We will throw error if unable to get tableFormat as this is needed to create ConversionSource from SourceTable: https://github.com/apache/incubator-xtable/pull/599/files#diff-90350a03dbbf828930b2ad5b2629bb8c9da7aaa24b7f984600e03e807a185f7bR79



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1883960868


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConversionSource.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+
+import org.apache.xtable.conversion.SourceCatalog;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.spi.extractor.CatalogConversionSource;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+public class GlueCatalogConversionSource implements CatalogConversionSource {

Review Comment:
   @vinishjail97 I have created separate class for CatalogConversionSource implementation as this seems more cleaner. 
   Furthermore, `ExternalCatalog` is not exposed, not sure if that is intentional.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1884679984


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogSyncClient.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.time.ZonedDateTime;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.conversion.TargetCatalog;
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.CatalogSyncClient;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.CreateDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.DatabaseInput;
+import software.amazon.awssdk.services.glue.model.DeleteTableRequest;
+import software.amazon.awssdk.services.glue.model.EntityNotFoundException;
+import software.amazon.awssdk.services.glue.model.GetDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+@Log4j2
+public class GlueCatalogSyncClient implements CatalogSyncClient<Table> {
+
+  protected static final String GLUE_EXTERNAL_TABLE_TYPE = "EXTERNAL_TABLE";
+  private static final String TEMP_SUFFIX = "_temp";
+
+  protected final TargetCatalog targetCatalog;
+  @Getter protected final GlueClient glueClient;
+  @Getter protected final GlueCatalogConfig glueCatalogConfig;
+  @Getter protected final Configuration configuration;
+  @Getter protected final GlueSchemaExtractor schemaExtractor;
+  protected final IcebergGlueCatalogSyncOperations icebergGlueCatalogSyncOperations;

Review Comment:
   `IcebergGlueCatalogSyncOperations` shouldn't create the table IMO, it should return the requests or the glue table objects for a given `InternalTable` object. 
   
   The glueClient is not initialized differently for iceberg, hudi or delta but the requests being passed to it are different so these classes for each format should be focussed on generating those requests. Right now the relationship is still like a extending a base class. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1882693672


##########
xtable-core/src/main/java/org/apache/xtable/reflection/ReflectionUtils.java:
##########
@@ -53,6 +55,34 @@ public static <T> T createInstanceOfClass(String className, Object... constructo
     }
   }
 
+  public static <T> T createInstanceOfClassFromStaticMethod(

Review Comment:
   This will be used for loading custom credential providers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1882700829


##########
xtable-aws/src/main/java/org/apache/xtable/glue/GlueSchemaExtractor.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.glue;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import lombok.AccessLevel;
+import lombok.NoArgsConstructor;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.exception.SchemaExtractorException;
+import org.apache.xtable.model.schema.InternalField;
+import org.apache.xtable.model.schema.InternalSchema;
+
+import software.amazon.awssdk.services.glue.model.Column;
+import software.amazon.awssdk.services.glue.model.Table;
+
+@NoArgsConstructor(access = AccessLevel.PRIVATE)
+public class GlueSchemaExtractor {

Review Comment:
   TODO: Add java doc



##########
xtable-aws/src/main/java/org/apache/xtable/glue/IcebergGlueCatalogSyncClient.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.glue;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.conversion.TargetCatalog;
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.CreateTableRequest;
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+import software.amazon.awssdk.services.glue.model.UpdateTableRequest;
+
+public class IcebergGlueCatalogSyncClient extends GlueCatalogSyncClient {

Review Comment:
   TODO: Add java doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1883966663


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/IcebergGlueCatalogSyncOperations.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.CreateTableRequest;
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+import software.amazon.awssdk.services.glue.model.UpdateTableRequest;
+
+class IcebergGlueCatalogSyncOperations {

Review Comment:
   Iceberg specific glue operations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1886700953


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogSyncClient.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.time.ZonedDateTime;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.conversion.TargetCatalog;
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.CatalogSyncClient;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.CreateDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.DatabaseInput;
+import software.amazon.awssdk.services.glue.model.DeleteTableRequest;
+import software.amazon.awssdk.services.glue.model.EntityNotFoundException;
+import software.amazon.awssdk.services.glue.model.GetDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+@Log4j2
+public class GlueCatalogSyncClient implements CatalogSyncClient<Table> {
+
+  protected static final String GLUE_EXTERNAL_TABLE_TYPE = "EXTERNAL_TABLE";
+  private static final String TEMP_SUFFIX = "_temp";
+
+  protected final TargetCatalog targetCatalog;
+  @Getter protected final GlueClient glueClient;
+  @Getter protected final GlueCatalogConfig glueCatalogConfig;
+  @Getter protected final Configuration configuration;
+  @Getter protected final GlueSchemaExtractor schemaExtractor;
+  protected final IcebergGlueCatalogSyncOperations icebergGlueCatalogSyncOperations;

Review Comment:
   Makes sense, updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1899861727


##########
xtable-core/src/main/java/org/apache/xtable/catalog/ExternalCatalogConfigFactory.java:
##########
@@ -20,16 +20,27 @@
 
 import java.util.Map;
 
+import org.apache.xtable.catalog.glue.GlueCatalogConversionSource;
+import org.apache.xtable.catalog.glue.GlueCatalogSyncClient;
 import org.apache.xtable.conversion.ExternalCatalogConfig;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.CatalogType;
 
 /** A factory class which returns {@link ExternalCatalogConfig} based on catalogType. */
 public class ExternalCatalogConfigFactory {
 
   public static ExternalCatalogConfig fromCatalogType(
       String catalogType, String catalogId, Map<String, String> properties) {
-    // TODO: Choose existing implementation based on catalogType.
-    String catalogSyncClientImpl = "";
-    String catalogConversionSourceImpl = "";
+    String catalogSyncClientImpl;
+    String catalogConversionSourceImpl;
+    switch (catalogType) {
+      case CatalogType.GLUE:
+        catalogSyncClientImpl = GlueCatalogSyncClient.class.getName();

Review Comment:
   We won't have these classes on the path once the code is broken out into modules, we'll want to adopt a similar pattern using the service loader like we did for the conversion targets



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900181081


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Yes. Btw, this property should be set automatically by the engine (which created the table) in most of the cases
   
   Does this javadoc make sense - 
   
   <img width="687" alt="image" src="https://github.com/user-attachments/assets/11338790-5d47-45ed-b841-483d0bb5eb4c" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900179574


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(

Review Comment:
   Instead of throwing exception, I can default to tableLocation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 closed pull request #599: [590] Add Iceberg Glue Catalog Sync implementation
URL: https://github.com/apache/incubator-xtable/pull/599


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1898596528


##########
xtable-core/pom.xml:
##########
@@ -138,12 +138,23 @@
             <scope>provided</scope>
         </dependency>
 
+        <!-- AWS Glue dependencies -->

Review Comment:
   I think it would be best to create a new module for the Glue related code. That will allow our users to only pull in the dependencies that they require.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900180545


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(

Review Comment:
   I think that is fine for now, wondering if there is a better place for this logic though so if someone adds an implementation that they can override this logic to fit their needs without needing to make core changes to XTable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1883965112


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogSyncClient.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.time.ZonedDateTime;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.conversion.TargetCatalog;
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.CatalogSyncClient;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.CreateDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.DatabaseInput;
+import software.amazon.awssdk.services.glue.model.DeleteTableRequest;
+import software.amazon.awssdk.services.glue.model.EntityNotFoundException;
+import software.amazon.awssdk.services.glue.model.GetDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+@Log4j2
+public class GlueCatalogSyncClient implements CatalogSyncClient<Table> {
+
+  protected static final String GLUE_EXTERNAL_TABLE_TYPE = "EXTERNAL_TABLE";
+  private static final String TEMP_SUFFIX = "_temp";
+
+  protected final TargetCatalog targetCatalog;
+  @Getter protected final GlueClient glueClient;
+  @Getter protected final GlueCatalogConfig glueCatalogConfig;
+  @Getter protected final Configuration configuration;
+  @Getter protected final GlueSchemaExtractor schemaExtractor;
+  protected final IcebergGlueCatalogSyncOperations icebergGlueCatalogSyncOperations;

Review Comment:
   @vinishjail97 Is this fine or you want some sort of manager / factory here? For factory, we might need to pass table format as targetCatalog does not have table format



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1884671092


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConversionSource.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+
+import org.apache.xtable.conversion.SourceCatalog;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.spi.extractor.CatalogConversionSource;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+public class GlueCatalogConversionSource implements CatalogConversionSource {

Review Comment:
   I got rid of SourceCatalog and TargetCatalog, simplified the catalog configuration, now I think it's just adding this function `SourceTable getSourceTable(CatalogTableIdentifier tableIdentifier)` to the impl class which implements this. 
   
   Regardless I will keep the config for `sourceCatalogConversionImplClass` available in OSS for users. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1901295672


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   yes, specifically an iceberg table managed with the glue catalog https://iceberg.apache.org/docs/1.6.0/aws/?h=glue#glue-catalog



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1882700225


##########
xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {

Review Comment:
   TODO: Add java doc



##########
xtable-aws/src/main/java/org/apache/xtable/glue/GlueCatalogSyncClient.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.glue;
+
+import java.time.ZonedDateTime;
+
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.xtable.conversion.TargetCatalog;
+import org.apache.xtable.exception.CatalogSyncException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.spi.sync.CatalogSyncClient;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.CreateDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.DatabaseInput;
+import software.amazon.awssdk.services.glue.model.DeleteTableRequest;
+import software.amazon.awssdk.services.glue.model.EntityNotFoundException;
+import software.amazon.awssdk.services.glue.model.GetDatabaseRequest;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+@Log4j2
+public abstract class GlueCatalogSyncClient implements CatalogSyncClient<Table> {

Review Comment:
   TODO: Add java doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1883958245


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {
+
+  private static final ObjectMapper OBJECT_MAPPER =
+      new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+  public static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX =
+      "externalCatalog.glue.credentials.provider.";
+
+  @JsonProperty("externalCatalog.glue.catalogId")
+  private String catalogId;
+
+  @JsonProperty("externalCatalog.glue.region")
+  private String region;
+
+  @JsonProperty("externalCatalog.glue.credentialsProviderClass")
+  private String clientCredentialsProviderClass;
+
+  /**
+   * In case a credentialsProviderClass is configured and require additional properties for
+   * instantiation, those properties should start with {@link #CLIENT_CREDENTIAL_PROVIDER_PREFIX}.
+   *
+   * <p>For ex: if credentialsProviderClass requires `accessKey` and `secretAccessKey`, they should
+   * be configured using below keys:
+   * <li>externalCatalog.glue.credentials.provider.accessKey
+   * <li>externalCatalog.glue.credentials.provider.secretAccessKey
+   */
+  @Setter private Map<String, String> clientCredentialConfigs;
+
+  public static GlueCatalogConfig of(Map<String, String> properties) {

Review Comment:
   @vinishjail97 Is this approach for creating config is fine or we should use a factory for creating these?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900196228


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/** Configurations for setting up Glue client and running Glue catalog operations */
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {
+
+  private static final ObjectMapper OBJECT_MAPPER =
+      new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+  public static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX =
+      "externalCatalog.glue.credentials.provider.";
+
+  @JsonProperty("externalCatalog.glue.catalogId")
+  private String catalogId;
+
+  @JsonProperty("externalCatalog.glue.region")
+  private String region;
+
+  @JsonProperty("externalCatalog.glue.credentialsProviderClass")
+  private String clientCredentialsProviderClass;
+
+  /**
+   * In case a credentialsProviderClass is configured and require additional properties for
+   * instantiation, those properties should start with {@link #CLIENT_CREDENTIAL_PROVIDER_PREFIX}.
+   *
+   * <p>For ex: if credentialsProviderClass requires `accessKey` and `secretAccessKey`, they should
+   * be configured using below keys:
+   * <li>externalCatalog.glue.credentials.provider.accessKey
+   * <li>externalCatalog.glue.credentials.provider.secretAccessKey
+   */
+  @Setter private Map<String, String> clientCredentialConfigs;
+
+  /** Creates GlueCatalogConfig from given key-value map */
+  public static GlueCatalogConfig of(Map<String, String> properties) {
+    try {
+      GlueCatalogConfig glueCatalogConfig =
+          OBJECT_MAPPER.readValue(
+              OBJECT_MAPPER.writeValueAsString(properties), GlueCatalogConfig.class);
+      Map<String, String> clientCredentialProperties =
+          propertiesWithPrefix(properties, CLIENT_CREDENTIAL_PROVIDER_PREFIX);
+      glueCatalogConfig.setClientCredentialConfigs(clientCredentialProperties);

Review Comment:
   Will check if this can be parsed directly using jackson



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1899922000


##########
pom.xml:
##########
@@ -372,9 +373,9 @@
                 <scope>runtime</scope>
             </dependency>
             <dependency>
-                <groupId>com.amazonaws</groupId>
-                <artifactId>aws-java-sdk-bundle</artifactId>
-                <version>1.12.328</version>
+                <groupId>software.amazon.awssdk</groupId>

Review Comment:
   We should define the `glue` artifact and version here in the dependencyManagement section as well



##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/** Configurations for setting up Glue client and running Glue catalog operations */
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {
+
+  private static final ObjectMapper OBJECT_MAPPER =
+      new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+  public static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX =
+      "externalCatalog.glue.credentials.provider.";
+
+  @JsonProperty("externalCatalog.glue.catalogId")
+  private String catalogId;
+
+  @JsonProperty("externalCatalog.glue.region")
+  private String region;
+
+  @JsonProperty("externalCatalog.glue.credentialsProviderClass")
+  private String clientCredentialsProviderClass;
+
+  /**
+   * In case a credentialsProviderClass is configured and require additional properties for
+   * instantiation, those properties should start with {@link #CLIENT_CREDENTIAL_PROVIDER_PREFIX}.
+   *
+   * <p>For ex: if credentialsProviderClass requires `accessKey` and `secretAccessKey`, they should
+   * be configured using below keys:
+   * <li>externalCatalog.glue.credentials.provider.accessKey
+   * <li>externalCatalog.glue.credentials.provider.secretAccessKey
+   */
+  @Setter private Map<String, String> clientCredentialConfigs;
+
+  /** Creates GlueCatalogConfig from given key-value map */
+  public static GlueCatalogConfig of(Map<String, String> properties) {
+    try {
+      GlueCatalogConfig glueCatalogConfig =
+          OBJECT_MAPPER.readValue(

Review Comment:
   Can you just use `convertValue` for this instead of writing out to an intermediate string?



##########
xtable-core/src/main/java/org/apache/xtable/catalog/CatalogUtils.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+
+public class CatalogUtils {
+
+  public static HierarchicalTableIdentifier castToHierarchicalTableIdentifier(

Review Comment:
   Let's add a basic unit test for this?



##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/** Configurations for setting up Glue client and running Glue catalog operations */
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {
+
+  private static final ObjectMapper OBJECT_MAPPER =
+      new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+  public static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX =
+      "externalCatalog.glue.credentials.provider.";
+
+  @JsonProperty("externalCatalog.glue.catalogId")
+  private String catalogId;
+
+  @JsonProperty("externalCatalog.glue.region")
+  private String region;
+
+  @JsonProperty("externalCatalog.glue.credentialsProviderClass")
+  private String clientCredentialsProviderClass;
+
+  /**
+   * In case a credentialsProviderClass is configured and require additional properties for
+   * instantiation, those properties should start with {@link #CLIENT_CREDENTIAL_PROVIDER_PREFIX}.
+   *
+   * <p>For ex: if credentialsProviderClass requires `accessKey` and `secretAccessKey`, they should
+   * be configured using below keys:
+   * <li>externalCatalog.glue.credentials.provider.accessKey
+   * <li>externalCatalog.glue.credentials.provider.secretAccessKey
+   */
+  @Setter private Map<String, String> clientCredentialConfigs;
+
+  /** Creates GlueCatalogConfig from given key-value map */
+  public static GlueCatalogConfig of(Map<String, String> properties) {
+    try {
+      GlueCatalogConfig glueCatalogConfig =
+          OBJECT_MAPPER.readValue(
+              OBJECT_MAPPER.writeValueAsString(properties), GlueCatalogConfig.class);
+      Map<String, String> clientCredentialProperties =
+          propertiesWithPrefix(properties, CLIENT_CREDENTIAL_PROVIDER_PREFIX);
+      glueCatalogConfig.setClientCredentialConfigs(clientCredentialProperties);

Review Comment:
   Is there a way to just do this parsing with jackson? Similar to what was done here: https://github.com/apache/incubator-xtable/commit/f6767883f8c27c4ad39e4bb54352fcdae964c11d#diff-ac70a3d50d9eed96fd75f20baf8289cddec6834b15ad6165c077769c3729760dR287



##########
xtable-core/src/main/java/org/apache/xtable/catalog/CatalogUtils.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+
+public class CatalogUtils {

Review Comment:
   Add a `@NoArgsConstructor(access = Private)` if this is meant to only expose methods and not allow developers to create instances



##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/DefaultGlueClientFactory.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.xtable.reflection.ReflectionUtils;
+
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.GlueClientBuilder;
+
+/**
+ * Factory class for creating and configuring instances of {@link GlueClient} with settings provided
+ * by {@link GlueCatalogConfig}.
+ *
+ * <p>This factory is responsible for setting the AWS region and credentials for the Glue client. If
+ * a custom credentials provider class is specified in {@code GlueCatalogConfig}, it will use
+ * reflection to instantiate the provider; otherwise, it defaults to the standard AWS credentials
+ * provider.
+ */
+public class DefaultGlueClientFactory extends GlueClientFactory {
+
+  public DefaultGlueClientFactory(GlueCatalogConfig glueConfig) {
+    super(glueConfig);
+  }
+
+  public GlueClient getGlueClient() {
+    GlueClientBuilder builder = GlueClient.builder();
+    if (!StringUtils.isEmpty(glueConfig.getRegion())) {
+      builder.region(Region.of(glueConfig.getRegion()));
+    }
+
+    AwsCredentialsProvider credentialsProvider;
+    if (!StringUtils.isEmpty(glueConfig.getClientCredentialsProviderClass())) {
+      String className = glueConfig.getClientCredentialsProviderClass();
+      try {
+        credentialsProvider =
+            ReflectionUtils.createInstanceOfClassFromStaticMethod(
+                className,
+                "create",
+                new Class<?>[] {Map.class},
+                new Object[] {glueConfig.getClientCredentialConfigs()});
+      } catch (Exception e) {

Review Comment:
   Can you define more specific exceptions to catch here? 



##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   Will this work with an external catalog for Iceberg?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900195996


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   what do you mean by "external catalog for iceberg" in this context?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900195614


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   - As per the documentation: https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables-syncing-metadata.html, Athena and AWS Glue consoles requires `spark.sql.sources.provider` property to be set when creating delta lake table using those methods.
   
   - We will throw error if unable to get tableFormat as I believe this is needed to create ConversionSource from SourceTable: https://github.com/apache/incubator-xtable/pull/599/files#diff-90350a03dbbf828930b2ad5b2629bb8c9da7aaa24b7f984600e03e807a185f7bR79



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900951086


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   Has this been tested? I did not think this would work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1901042581


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   > source: Iceberg table with glue catalog, target: Iceberg HMS
   
   Is this a right scenario to test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1886699904


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConversionSource.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+
+import org.apache.xtable.conversion.SourceCatalog;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.spi.extractor.CatalogConversionSource;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+public class GlueCatalogConversionSource implements CatalogConversionSource {

Review Comment:
   sure, updated GlueCatalogSyncClient to implement `getSourceTable `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1882698682


##########
xtable-aws/pom.xml:
##########
@@ -0,0 +1,96 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.xtable</groupId>
+        <artifactId>xtable</artifactId>
+        <version>0.2.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>xtable-aws</artifactId>
+    <name>XTable Project Core</name>

Review Comment:
   TODO: Update name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1883961758


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConversionSource.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+
+import org.apache.xtable.conversion.SourceCatalog;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.spi.extractor.CatalogConversionSource;
+
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.GetTableRequest;
+import software.amazon.awssdk.services.glue.model.GetTableResponse;
+import software.amazon.awssdk.services.glue.model.Table;
+
+public class GlueCatalogConversionSource implements CatalogConversionSource {
+
+  private static final String TABLE_TYPE_PROP = "table_type";
+  private final SourceCatalog sourceCatalog;
+  private final GlueCatalogConfig glueCatalogConfig;
+  private final GlueClient glueClient;
+
+  public GlueCatalogConversionSource(SourceCatalog sourceCatalog, Configuration configuration) {
+    this.sourceCatalog = sourceCatalog;
+    this.glueCatalogConfig =
+        GlueCatalogConfig.of(sourceCatalog.getCatalogConfig().getCatalogOptions());
+    this.glueClient = new DefaultGlueClientFactory(glueCatalogConfig).getGlueClient();
+  }
+
+  @VisibleForTesting
+  GlueCatalogConversionSource(
+      SourceCatalog sourceCatalog, GlueClient glueClient, GlueCatalogConfig glueCatalogConfig) {
+    this.sourceCatalog = sourceCatalog;
+    this.glueCatalogConfig = glueCatalogConfig;
+    this.glueClient = glueClient;
+  }
+
+  @Override
+  public SourceTable getSourceTable(CatalogTableIdentifier tableIdentifier) {

Review Comment:
   TODO: Add tests



##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {

Review Comment:
   TODO: Add tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900602764


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   Got it. There should not be any issues if an external catalog is configured as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900603082


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Have moved the null check inside the method 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1884671231


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/GlueCatalogConfig.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+@Getter
+@EqualsAndHashCode
+@ToString
+public class GlueCatalogConfig {
+
+  private static final ObjectMapper OBJECT_MAPPER =
+      new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+  public static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX =
+      "externalCatalog.glue.credentials.provider.";
+
+  @JsonProperty("externalCatalog.glue.catalogId")
+  private String catalogId;
+
+  @JsonProperty("externalCatalog.glue.region")
+  private String region;
+
+  @JsonProperty("externalCatalog.glue.credentialsProviderClass")
+  private String clientCredentialsProviderClass;
+
+  /**
+   * In case a credentialsProviderClass is configured and require additional properties for
+   * instantiation, those properties should start with {@link #CLIENT_CREDENTIAL_PROVIDER_PREFIX}.
+   *
+   * <p>For ex: if credentialsProviderClass requires `accessKey` and `secretAccessKey`, they should
+   * be configured using below keys:
+   * <li>externalCatalog.glue.credentials.provider.accessKey
+   * <li>externalCatalog.glue.credentials.provider.secretAccessKey
+   */
+  @Setter private Map<String, String> clientCredentialConfigs;
+
+  public static GlueCatalogConfig of(Map<String, String> properties) {

Review Comment:
   This is okay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900205988


##########
xtable-utilities/src/main/resources/xtable-hadoop-defaults.xml:
##########
@@ -55,15 +55,15 @@
   </property>
   <property>
     <name>fs.s3.aws.credentials.provider</name>
-    <value>com.amazonaws.auth.DefaultAWSCredentialsProviderChain</value>
+    <value>software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider</value>

Review Comment:
   PR for aws sdk and hadoop version upgrade: https://github.com/apache/incubator-xtable/pull/614



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900241634


##########
xtable-core/src/main/java/org/apache/xtable/catalog/glue/table/IcebergGlueCatalogTableBuilder.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog.glue.table;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.PREVIOUS_METADATA_LOCATION_PROP;
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.CatalogUtils.castToHierarchicalTableIdentifier;
+import static org.apache.xtable.catalog.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.catalog.CatalogTableBuilder;
+import org.apache.xtable.catalog.glue.GlueSchemaExtractor;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.catalog.CatalogTableIdentifier;
+import org.apache.xtable.model.catalog.HierarchicalTableIdentifier;
+import org.apache.xtable.model.storage.TableFormat;
+
+import software.amazon.awssdk.services.glue.model.StorageDescriptor;
+import software.amazon.awssdk.services.glue.model.Table;
+import software.amazon.awssdk.services.glue.model.TableInput;
+
+/** Iceberg specific table operations for Glue catalog sync */
+public class IcebergGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {
+
+  private final GlueSchemaExtractor schemaExtractor;
+  private final HadoopTables hadoopTables;
+  private static final String tableFormat = TableFormat.ICEBERG;
+
+  public IcebergGlueCatalogTableBuilder(Configuration configuration) {
+    this.schemaExtractor = GlueSchemaExtractor.getInstance();
+    this.hadoopTables = new HadoopTables(configuration);
+  }
+
+  @VisibleForTesting
+  IcebergGlueCatalogTableBuilder(GlueSchemaExtractor schemaExtractor, HadoopTables hadoopTables) {
+    this.schemaExtractor = schemaExtractor;
+    this.hadoopTables = hadoopTables;
+  }
+
+  @Override
+  public TableInput getCreateTableRequest(
+      InternalTable table, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable fsTable = loadTableFromFs(table.getBasePath());
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(getTableParameters(fsTable))
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(schemaExtractor.toColumns(tableFormat, table.getReadSchema()))
+                .build())
+        .build();
+  }
+
+  @Override
+  public TableInput getUpdateTableRequest(
+      InternalTable table, Table catalogTable, CatalogTableIdentifier tableIdentifier) {
+    HierarchicalTableIdentifier tblIdentifier = castToHierarchicalTableIdentifier(tableIdentifier);
+    BaseTable icebergTable = loadTableFromFs(table.getBasePath());
+    Map<String, String> parameters = new HashMap<>(catalogTable.parameters());
+    parameters.put(PREVIOUS_METADATA_LOCATION_PROP, parameters.get(METADATA_LOCATION_PROP));
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    parameters.putAll(icebergTable.properties());
+
+    return TableInput.builder()
+        .name(tblIdentifier.getTableName())
+        .tableType(GLUE_EXTERNAL_TABLE_TYPE)
+        .parameters(parameters)
+        .storageDescriptor(
+            StorageDescriptor.builder()
+                .location(table.getBasePath())
+                .columns(
+                    schemaExtractor.toColumns(tableFormat, table.getReadSchema(), catalogTable))
+                .build())
+        .build();
+  }
+
+  @VisibleForTesting
+  Map<String, String> getTableParameters(BaseTable icebergTable) {
+    Map<String, String> parameters = new HashMap<>(icebergTable.properties());
+    parameters.put(TABLE_TYPE_PROP, tableFormat);
+    parameters.put(METADATA_LOCATION_PROP, getMetadataFileLocation(icebergTable));
+    return parameters;
+  }
+
+  private BaseTable loadTableFromFs(String tableBasePath) {

Review Comment:
   Production tables for Iceberg are managed with a catalog that handles the transactions, especially when dealing with multiple writer cases. This is why we allow users to specify that configuration when using Iceberg as a source or target for conversion today. In the conversion, we use that catalog to get the state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [590] Add Iceberg Glue Catalog Sync implementation [incubator-xtable]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #599:
URL: https://github.com/apache/incubator-xtable/pull/599#discussion_r1900241845


##########
xtable-core/src/main/java/org/apache/xtable/catalog/TableFormatUtils.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.catalog;
+
+import static org.apache.iceberg.BaseMetastoreTableOperations.TABLE_TYPE_PROP;
+import static org.apache.xtable.catalog.Constants.PROP_SPARK_SQL_SOURCES_PROVIDER;
+
+import java.util.Map;
+
+import org.apache.iceberg.TableProperties;
+
+import com.google.common.base.Strings;
+
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.model.storage.TableFormat;
+
+public class TableFormatUtils {
+
+  public static String getTableDataLocation(
+      String tableFormat, String tableLocation, Map<String, String> properties) {
+    switch (tableFormat) {
+      case TableFormat.ICEBERG:
+        return getIcebergDataLocation(tableLocation, properties);
+      case TableFormat.DELTA:
+      case TableFormat.HUDI:
+        return tableLocation;
+      default:
+        throw new NotSupportedException("Unsupported table format: " + tableFormat);
+    }
+  }
+
+  /** Get iceberg table data files location */
+  private static String getIcebergDataLocation(
+      String tableLocation, Map<String, String> properties) {
+    String dataLocation = properties.get(TableProperties.WRITE_DATA_LOCATION);
+    if (dataLocation == null) {
+      dataLocation = properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+      if (dataLocation == null) {
+        dataLocation = properties.get(TableProperties.OBJECT_STORE_PATH);
+        if (dataLocation == null) {
+          dataLocation = String.format("%s/data", tableLocation);
+        }
+      }
+    }
+    return dataLocation;
+  }
+
+  // Get table format name from table properties

Review Comment:
   Does it make more sense to move that logic into the method instead of making each caller throw that exception? 
   
   If you think it makes more sense to force the caller to handle it, let's consider using Optional so it is clear the value may not be set and the caller must handle it appropriately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org