You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/12/07 18:41:01 UTC

[GitHub] [iceberg] rymurr opened a new pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

rymurr opened a new pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537760489



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);

Review comment:
       We usually make `Splitter` and `Joiner` instances static fields.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r538848436



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +193,165 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties) {
+    return buildTable(location, schema).withPartitionSpec(spec).withProperties(properties).createTransaction();
+  }
+
+  /**
+   * Start a transaction to replace a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @param orCreate whether to create the table if not exists
+   * @return a {@link Transaction} to replace the table
+   * @throws NoSuchTableException if the table doesn't exist and orCreate is false
+   */
+  public Transaction newReplaceTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties,
+      boolean orCreate) {
+
+
+    Catalog.TableBuilder builder = buildTable(location, schema).withPartitionSpec(spec).withProperties(properties);
+    return orCreate ? builder.createOrReplaceTransaction() : builder.replaceTransaction();
+  }
+
+  public Catalog.TableBuilder buildTable(String location, Schema schema) {
+    return new HadoopTableBuilder(location, schema);
+  }
+
+  private class HadoopTableBuilder implements Catalog.TableBuilder {
+    private final String location;
+    private final Schema schema;
+    private final ImmutableMap.Builder<String, String> propertiesBuilder = ImmutableMap.builder();
+    private PartitionSpec spec = PartitionSpec.unpartitioned();
+    private SortOrder sortOrder = SortOrder.unsorted();
+
+
+    HadoopTableBuilder(String location, Schema schema) {
+      this.location = location;
+      this.schema = schema;
+    }
+
+    @Override
+    public Catalog.TableBuilder withPartitionSpec(PartitionSpec newSpec) {
+      this.spec = newSpec != null ? newSpec : PartitionSpec.unpartitioned();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withSortOrder(SortOrder newSortOrder) {
+      this.sortOrder = newSortOrder != null ? newSortOrder : SortOrder.unsorted();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withLocation(String newLocation) {
+      Preconditions.checkArgument(newLocation == null || location.equals(newLocation),
+          String.format("Table location %s differs from the table location (%s) from the PathIdentifier",
+              newLocation, location));

Review comment:
       Nit: preconditions already support argument formatting, so `String.format` is redundant.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536930840



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +197,76 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param identifier a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(

Review comment:
       Can this use `TableBuilder` instead? That would make the `SparkCatalog` implementation a lot cleaner because the code to configure the builder could be shared.
   
   I opened https://github.com/apache/iceberg/pull/1879 because tests would fail unless the builder is also supported in CachingCatalog.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537761984



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ?
+        new String[]{Joiner.on("/").join(pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]))} :

Review comment:
       yup, hangover from previous change. Fixed now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740158245


   > What are the plans for a test suite on this?
   
   We can always add a suite that passes `PathIdentifier` to a catalog. I was also thinking that once this is included with #1783, all of the existing tests for IcebergSource with path tables will be applied.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536633817



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -132,10 +134,17 @@ protected TableIdentifier buildIdentifier(Identifier identifier) {
     return TableIdentifier.of(Namespace.of(identifier.namespace()), identifier.name());
   }
 
+  private String[] currentNamespace() {
+    return SparkSession.active().sessionState().catalogManager().currentNamespace();

Review comment:
       fixed. Just for my learning why is it ok to get hadoop config from active session but not the current namespace? I have been following the commonly used `SparkSession.active().sessionState().newHadoopConf()` pattern




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537738542



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +193,162 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties) {
+    return buildTable(location, schema).withPartitionSpec(spec).withProperties(properties).createTransaction();
+  }
+
+  /**
+   * Start a transaction to replace a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @param orCreate whether to create the table if not exists
+   * @return a {@link Transaction} to replace the table
+   * @throws NoSuchTableException if the table doesn't exist and orCreate is false
+   */
+  public Transaction newReplaceTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties,
+      boolean orCreate) {
+
+
+    Catalog.TableBuilder builder = buildTable(location, schema).withPartitionSpec(spec).withProperties(properties);
+    return orCreate ? builder.createOrReplaceTransaction() : builder.replaceTransaction();
+  }
+
+  public Catalog.TableBuilder buildTable(String location, Schema schema) {
+    return new HadoopTableBuilder(location, schema);
+  }
+
+  private class HadoopTableBuilder implements Catalog.TableBuilder {
+    private final String location;
+    private final Schema schema;
+    private final ImmutableMap.Builder<String, String> propertiesBuilder = ImmutableMap.builder();
+    private PartitionSpec spec = PartitionSpec.unpartitioned();
+    private SortOrder sortOrder = SortOrder.unsorted();
+
+
+    HadoopTableBuilder(String location, Schema schema) {
+      this.location = location;
+      this.schema = schema;
+    }
+
+    @Override
+    public Catalog.TableBuilder withPartitionSpec(PartitionSpec newSpec) {
+      this.spec = newSpec != null ? newSpec : PartitionSpec.unpartitioned();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withSortOrder(SortOrder newSortOrder) {
+      this.sortOrder = newSortOrder != null ? newSortOrder : SortOrder.unsorted();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withLocation(String newLocation) {
+      return this;

Review comment:
       I think this should validate that the location matches the one passed to create the builder. If this is a different location, it can't just be ignored.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537765366



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -268,14 +329,19 @@ public void renameTable(Identifier from, Identifier to) throws NoSuchTableExcept
   @Override
   public void invalidateTable(Identifier ident) {
     try {
-      icebergCatalog.loadTable(buildIdentifier(ident)).refresh();
+      TableIdentifier tableIdentifier = buildIdentifier(ident);
+      Table table = isPathIdentifier(ident) ?
+          tables.load(tableIdentifier.name()) :
+          icebergCatalog.loadTable(tableIdentifier);
+      table.refresh();
     } catch (org.apache.iceberg.exceptions.NoSuchTableException ignored) {
       // ignore if the table doesn't exist, it is not cached
     }
   }
 
   @Override
   public Identifier[] listTables(String[] namespace) {
+    // no way to identify if this is a path and we should use tables instead of catalog.

Review comment:
       Ah I see, have removed the comment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537384242



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -257,6 +317,7 @@ public boolean dropTable(Identifier ident) {
   @Override
   public void renameTable(Identifier from, Identifier to) throws NoSuchTableException, TableAlreadyExistsException {
     try {
+      // can't rename hadoop tables

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734867957


   > I was thinking about this a bit differently, rather than providing a full catalog I was thinking that we just have the various spark catalogs treat file types as namespaces. Then we just have them switch to returning Hadoop tables in the load table method.
   
   I was thinking this way originally as well. My first attempt at it made for weird interactions between `BaseMetastoreCatalog` and the individual catalogs. And a bunch of dupe code in the various `Catalog` impls. I am still not satisfied w/ this solution as the hadoop delegate catalog is a bit strange but at least it centralises the logic a bit.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537388733



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ? pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]) :
+        new String[0];

Review comment:
       agreed that it shouldn't be `["s3:", "", "bucket", "path", "to"]` but I am not sure why it would be the full location. Then we have `namespace` returning `{location}` the new public method `location` returning `location` and the `name` method returning just the filename. To me `namespace` + `name` should equal location. I have updated this patch w/ my suggestion but happy to revert.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734852303


   I was thinking about this a bit differently, rather than providing a full catalog I was thinking that we just have the various spark catalogs treat file types as namespaces. Then we just have them switch to returning Hadoop tables in the load table method


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740133380


   @RussellSpitzer Ive been wondering the same. It is hard to test till #1783 is merged. I will have a think tho
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r550101676



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +193,165 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties) {
+    return buildTable(location, schema).withPartitionSpec(spec).withProperties(properties).createTransaction();
+  }
+
+  /**
+   * Start a transaction to replace a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @param orCreate whether to create the table if not exists
+   * @return a {@link Transaction} to replace the table
+   * @throws NoSuchTableException if the table doesn't exist and orCreate is false
+   */
+  public Transaction newReplaceTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties,
+      boolean orCreate) {
+
+
+    Catalog.TableBuilder builder = buildTable(location, schema).withPartitionSpec(spec).withProperties(properties);
+    return orCreate ? builder.createOrReplaceTransaction() : builder.replaceTransaction();
+  }
+
+  public Catalog.TableBuilder buildTable(String location, Schema schema) {
+    return new HadoopTableBuilder(location, schema);
+  }
+
+  private class HadoopTableBuilder implements Catalog.TableBuilder {
+    private final String location;
+    private final Schema schema;
+    private final ImmutableMap.Builder<String, String> propertiesBuilder = ImmutableMap.builder();
+    private PartitionSpec spec = PartitionSpec.unpartitioned();
+    private SortOrder sortOrder = SortOrder.unsorted();
+
+
+    HadoopTableBuilder(String location, Schema schema) {
+      this.location = location;
+      this.schema = schema;
+    }
+
+    @Override
+    public Catalog.TableBuilder withPartitionSpec(PartitionSpec newSpec) {
+      this.spec = newSpec != null ? newSpec : PartitionSpec.unpartitioned();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withSortOrder(SortOrder newSortOrder) {
+      this.sortOrder = newSortOrder != null ? newSortOrder : SortOrder.unsorted();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withLocation(String newLocation) {
+      Preconditions.checkArgument(newLocation == null || location.equals(newLocation),
+          String.format("Table location %s differs from the table location (%s) from the PathIdentifier",
+              newLocation, location));
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withProperties(Map<String, String> properties) {
+      if (properties != null) {
+        propertiesBuilder.putAll(properties);
+      }
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withProperty(String key, String value) {
+      propertiesBuilder.put(key, value);
+      return this;
+    }
+
+    @Override
+    public Table create() {
+      TableOperations ops = newTableOps(location);
+      if (ops.current() != null) {
+        throw new AlreadyExistsException("Table already exists at location: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata = tableMetadata(schema, spec, sortOrder, properties, location);
+      ops.commit(null, metadata);
+      return new BaseTable(ops, location);
+    }
+
+    @Override
+    public Transaction createTransaction() {
+      TableOperations ops = newTableOps(location);
+      if (ops.current() != null) {
+        throw new AlreadyExistsException("Table already exists: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata = tableMetadata(schema, spec, null, properties, location);
+      return Transactions.createTableTransaction(location, ops, metadata);
+    }
+
+    @Override
+    public Transaction replaceTransaction() {
+      return newReplaceTableTransaction(false);
+    }
+
+    @Override
+    public Transaction createOrReplaceTransaction() {
+      return newReplaceTableTransaction(true);
+    }
+
+    private Transaction newReplaceTableTransaction(boolean orCreate) {
+      TableOperations ops = newTableOps(location);
+      if (!orCreate && ops.current() == null) {
+        throw new NoSuchTableException("No such table: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata;
+      if (ops.current() != null) {
+        metadata = ops.current().buildReplacement(schema, spec, SortOrder.unsorted(), location, properties);
+      } else {
+        metadata = tableMetadata(schema, spec, null, properties, location);

Review comment:
       @rymurr  Here we use the `SortOrder.unsorted` or `null` to set the sort order rather than the `sortOrder` set from [here](https://github.com/apache/iceberg/pull/1843/files#diff-633b0a39dfee3393aee1717d9d83cd7e689d8672b24da82f809d8c21a933622dR271) ?  Is it correct ?  IMO,  we've ignored the user-provied sort order ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r533652664



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -165,6 +179,7 @@ public StagedTable stageCreate(Identifier ident, StructType schema, Transform[]
                                  Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
     try {
+      // can't stage a hadoop table

Review comment:
       After thinking about this a little more, I think we will need to support some form of staging for Hadoop tables. Because this catalog implements the atomic operation mix-in, the staging calls will be used for all CTAS plans. Using `SupportsCatalogOptions` would mean that `save()` gets turned into a CTAS. So if we don't want the existing creates to fail, we have to support a staged table.
   
   We can do that in a couple of ways. First, we could create a table builder based on `HadoopCatalogTableBuilder` that supports a location. Second, we could reuse the fake staged table from the session catalog (for non-Iceberg tables). I'd prefer to create a builder that can construct the transactions for path tables. We could add it to `HadoopTables`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-737921384


   Hey @jackye1995 thanks for the comments! The first approach in this patch was trying to tackle the problem with a new catalog. You can see it [here](https://github.com/apache/iceberg/pull/1843/commits/9cf2bede01a9194e730f78888914bd9c6570aee3). On the plus side it keeps all the filesystem handling in 1 class but it does have the downside of creating a lot of duplicate code and wasn't super nice. @RussellSpitzer suggested the approach in this PR and I think I prefer it now too, it keeps the scope if this change very small too.
   
   What I am struggling w/ at the moment is:
   * how to robustly identify a path in an `Identifier` (eg not rely solely on the presence of `/`)
   * how to correctly handle edge cases like ```catalog.namespace.`file://path/to/table` ```. What does a namespace mean in this case and how do we represent a path as an Identifier or TableIdentifier. 
   
   For the second point I think we should reject any namespace elements and treat a path as a namespaceless table with the full table path as the table/identifier name. For the first part I think the only reliable way is to try to do IO with the path: can we get an FS from the path, is it the local fs, is it a relative path etc.
   
   Thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537385129



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -268,14 +329,19 @@ public void renameTable(Identifier from, Identifier to) throws NoSuchTableExcept
   @Override
   public void invalidateTable(Identifier ident) {
     try {
-      icebergCatalog.loadTable(buildIdentifier(ident)).refresh();
+      TableIdentifier tableIdentifier = buildIdentifier(ident);
+      Table table = isPathIdentifier(ident) ?
+          tables.load(tableIdentifier.name()) :
+          icebergCatalog.loadTable(tableIdentifier);
+      table.refresh();
     } catch (org.apache.iceberg.exceptions.NoSuchTableException ignored) {
       // ignore if the table doesn't exist, it is not cached
     }
   }
 
   @Override
   public Identifier[] listTables(String[] namespace) {
+    // no way to identify if this is a path and we should use tables instead of catalog.

Review comment:
       not sure how to identify a path here? Single element w/ '/' in it? Seems arbitrary. The `HadoopCatalog` treats directory names as namespaces so there would be no way to identify that here.

##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -453,4 +520,5 @@ private static void commitChanges(Table table, SetProperty setLocation, SetPrope
 
     transaction.commitTransaction();
   }
+

Review comment:
       removed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-738252638


   > how to robustly identify a path in an Identifier (eg not rely solely on the presence of `/`)
   
   With all the given constraints, I think we have to rely on some string pattern to determine if it is a path or table. Use `/` is probably not a too bad choice considering that people won't normally put `/` in the table name. If we restrict the path to always be a full file path and do not allow relative path, then we can use a URI check (or IO check as you said for extra safety) to see if it is a Hadoop table. Is there a use case for relative path for Hadoop tables?
   
   And with whatever way we go for the check, I think it should be somewhere in the core package, instead of a protected method `isHadoopTable`, so that it can be used across compute engines for consistent behavior. One way to go is to add it as a method in `HadoopTables`.
   
   > how to correctly handle edge cases like catalog.namespace.`file://path/to/table`. What does a namespace mean in this case and how do we represent a path as an Identifier or TableIdentifier.
   
   +1 for 
   
   > reject any namespace elements and treat a path as a namespaceless table with the full table path as the table/identifier name


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536931516



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ? pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]) :
+        new String[0];

Review comment:
       I think that the namespace should be the location, not an array of path parts. This would create really weird namespaces, like `["s3:", "", "bucket", "path", "to"]`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r550161500



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +193,165 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties) {
+    return buildTable(location, schema).withPartitionSpec(spec).withProperties(properties).createTransaction();
+  }
+
+  /**
+   * Start a transaction to replace a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @param orCreate whether to create the table if not exists
+   * @return a {@link Transaction} to replace the table
+   * @throws NoSuchTableException if the table doesn't exist and orCreate is false
+   */
+  public Transaction newReplaceTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties,
+      boolean orCreate) {
+
+
+    Catalog.TableBuilder builder = buildTable(location, schema).withPartitionSpec(spec).withProperties(properties);
+    return orCreate ? builder.createOrReplaceTransaction() : builder.replaceTransaction();
+  }
+
+  public Catalog.TableBuilder buildTable(String location, Schema schema) {
+    return new HadoopTableBuilder(location, schema);
+  }
+
+  private class HadoopTableBuilder implements Catalog.TableBuilder {
+    private final String location;
+    private final Schema schema;
+    private final ImmutableMap.Builder<String, String> propertiesBuilder = ImmutableMap.builder();
+    private PartitionSpec spec = PartitionSpec.unpartitioned();
+    private SortOrder sortOrder = SortOrder.unsorted();
+
+
+    HadoopTableBuilder(String location, Schema schema) {
+      this.location = location;
+      this.schema = schema;
+    }
+
+    @Override
+    public Catalog.TableBuilder withPartitionSpec(PartitionSpec newSpec) {
+      this.spec = newSpec != null ? newSpec : PartitionSpec.unpartitioned();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withSortOrder(SortOrder newSortOrder) {
+      this.sortOrder = newSortOrder != null ? newSortOrder : SortOrder.unsorted();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withLocation(String newLocation) {
+      Preconditions.checkArgument(newLocation == null || location.equals(newLocation),
+          String.format("Table location %s differs from the table location (%s) from the PathIdentifier",
+              newLocation, location));
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withProperties(Map<String, String> properties) {
+      if (properties != null) {
+        propertiesBuilder.putAll(properties);
+      }
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withProperty(String key, String value) {
+      propertiesBuilder.put(key, value);
+      return this;
+    }
+
+    @Override
+    public Table create() {
+      TableOperations ops = newTableOps(location);
+      if (ops.current() != null) {
+        throw new AlreadyExistsException("Table already exists at location: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata = tableMetadata(schema, spec, sortOrder, properties, location);
+      ops.commit(null, metadata);
+      return new BaseTable(ops, location);
+    }
+
+    @Override
+    public Transaction createTransaction() {
+      TableOperations ops = newTableOps(location);
+      if (ops.current() != null) {
+        throw new AlreadyExistsException("Table already exists: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata = tableMetadata(schema, spec, null, properties, location);
+      return Transactions.createTableTransaction(location, ops, metadata);
+    }
+
+    @Override
+    public Transaction replaceTransaction() {
+      return newReplaceTableTransaction(false);
+    }
+
+    @Override
+    public Transaction createOrReplaceTransaction() {
+      return newReplaceTableTransaction(true);
+    }
+
+    private Transaction newReplaceTableTransaction(boolean orCreate) {
+      TableOperations ops = newTableOps(location);
+      if (!orCreate && ops.current() == null) {
+        throw new NoSuchTableException("No such table: %s", location);
+      }
+
+      Map<String, String> properties = propertiesBuilder.build();
+      TableMetadata metadata;
+      if (ops.current() != null) {
+        metadata = ops.current().buildReplacement(schema, spec, SortOrder.unsorted(), location, properties);
+      } else {
+        metadata = tableMetadata(schema, spec, null, properties, location);

Review comment:
       good spot @openinx not sure how that sneaked in! I have rasied #2009 to address




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536374139



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -132,10 +134,17 @@ protected TableIdentifier buildIdentifier(Identifier identifier) {
     return TableIdentifier.of(Namespace.of(identifier.namespace()), identifier.name());
   }
 
+  private String[] currentNamespace() {
+    return SparkSession.active().sessionState().catalogManager().currentNamespace();

Review comment:
       If this uses a session, then it should be identified when the catalog is created and stored as an instance field. Catalogs are specific to a SQL session, so they should not use the active session except for in initialization. After that, the same session should always be used.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537406133



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ? pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]) :
+        new String[0];

Review comment:
       Shouldn't namespace be just `iceberg` and name `location` to match built-in sources?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue merged pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537386230



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +197,76 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param identifier a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(

Review comment:
       fixed. Will rebase once #1879 is merged. I also prefer the `TableBuilder` pattern, much cleaner.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739081411


   > Considering the behavior for SQL is undefined, I would rather not add support for this right away. Delta needs this because it already supported these identifiers in v1 and has to be compatible. But I think we should have a use case that requires this before adding support.
   
   👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-736732550


   > In my mind this means the Spark3 tests that run against the session catalog are (partially) broken as the DROP command won't work for anything other than a Hive based table.
   
   I ran into this and ended up skipping non-Hive tests in other suites. We still have fairly good coverage because the session catalog is simply delegating. So as long as it works for a SparkCatalog backed by Hive, it will work for other catalog types.
   
   Fixing this in Spark would be great.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer edited a comment on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734852303


   I was thinking about this a bit differently, rather than providing a full catalog I was thinking that we just have the various spark catalogs treat file types as namespaces. Then we just have them switch to returning Hadoop tables in the load table method.
   
   Here is an example @aokolnychyi was typing up on the create ticket
   
   ```
   // Case 2.1: migrate a location using HadoopTables (or path-based catalog) for target
   
   MIGRATE TABLE parquet.`path/to/table`
   USING iceberg
   TBLPROPERTIES (
     'key' 'value'
   )
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739165535


   Thanks for the comments @rdblue, I have reduced the scope as suggested and updated the path identifier check.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536931701



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -164,13 +183,24 @@ public SparkTable createTable(Identifier ident, StructType schema,
   public StagedTable stageCreate(Identifier ident, StructType schema, Transform[] transforms,
                                  Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
+    TableIdentifier tableIdentifier = buildIdentifier(ident);
     try {
-      return new StagedSparkTable(icebergCatalog.newCreateTableTransaction(
-          buildIdentifier(ident),
-          icebergSchema,
-          Spark3Util.toPartitionSpec(icebergSchema, transforms),
-          properties.get("location"),
-          Spark3Util.rebuildCreateProperties(properties)));
+      Transaction transaction;
+      if (isPathIdentifier(ident)) {
+        transaction = tables.newCreateTableTransaction(
+            tableIdentifier.name(),

Review comment:
       This should be `tableIdentifier.location()` right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740102830


   just rebased w/ your caching catalog change @rdblue. Hopefully the build shouild be green now
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi edited a comment on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739840288


   Should we also respect `spark.sql.runSQLOnFiles` in Spark? I am not sure whether we should match built-in sources here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740151085


   Thanks @rdblue! both the check in HadoopTables and the simplification in SparkCatalog have been added.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536929634



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -268,14 +329,19 @@ public void renameTable(Identifier from, Identifier to) throws NoSuchTableExcept
   @Override
   public void invalidateTable(Identifier ident) {
     try {
-      icebergCatalog.loadTable(buildIdentifier(ident)).refresh();
+      TableIdentifier tableIdentifier = buildIdentifier(ident);
+      Table table = isPathIdentifier(ident) ?
+          tables.load(tableIdentifier.name()) :
+          icebergCatalog.loadTable(tableIdentifier);
+      table.refresh();
     } catch (org.apache.iceberg.exceptions.NoSuchTableException ignored) {
       // ignore if the table doesn't exist, it is not cached
     }
   }
 
   @Override
   public Identifier[] listTables(String[] namespace) {
+    // no way to identify if this is a path and we should use tables instead of catalog.

Review comment:
       There should also be no way to pass a path as a namespace.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-736749862


   I agree with the approach to use `HadoopTables` to return tables when a path identifier is found. What I'm not sure about is how to pass a path as an identifier. Supporting a namespace element that has the file format is useful for imports, but not for the `IcebergSource` because I don't think that we want the source to support those identifiers.
   
   For table imports, I think we have more options because we're controlling the parsed statement or the stored procedure. That procedure could use optional arguments like `file_format` or `location`. Similarly, a parsed statement can be transformed with rules to pass the reference as something other than an `Identifier`. What is relevant here is that we have options for those cases, so I think we should focus in this PR on how to pass paths from `IcebergSource` to a catalog, and loading in that catalog.
   
   Looking at the Spark, the identifier is immediately used to load the table, or is added to a CTAS plan. I don't think that the identifier is modified after it is returned. We could use a custom `Identifier` implementation, `PathIdentifier`, to signal that a path should be loaded.
   
   ```java
   public class PathIdentifier implements Identifier {
     private final String location;
     private final String name;
   
     public PathIdentifier(String location) {
       this.location = location;
       this.name = Iterables.getLastElement(Splitter.on("/").splitToList(location));
     }
   
     public String location() {
       return location;
     }
   
     public String namespace() {
       return new String[] { location };
     }
   
     public String name() {
       return name;
     }
   }
   ```
   
   This uses the last part of the location string as the table name, so that the default subquery aliases added in Spark work.
   
   Then each method in `SparkCatalog` would just need to check `instanceof PathIdentifier`, which is a bit cleaner than the `/` check -- although that would still need to be done in `IcebergSource`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537761946



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -268,14 +329,19 @@ public void renameTable(Identifier from, Identifier to) throws NoSuchTableExcept
   @Override
   public void invalidateTable(Identifier ident) {
     try {
-      icebergCatalog.loadTable(buildIdentifier(ident)).refresh();
+      TableIdentifier tableIdentifier = buildIdentifier(ident);
+      Table table = isPathIdentifier(ident) ?
+          tables.load(tableIdentifier.name()) :
+          icebergCatalog.loadTable(tableIdentifier);
+      table.refresh();
     } catch (org.apache.iceberg.exceptions.NoSuchTableException ignored) {
       // ignore if the table doesn't exist, it is not cached
     }
   }
 
   @Override
   public Identifier[] listTables(String[] namespace) {
+    // no way to identify if this is a path and we should use tables instead of catalog.

Review comment:
       Sorry, I wasn't very clear. If anything calls this, then it went through a path for normal identifiers. So I don't think that there is a need to worry about paths here because there isn't a way to pass a path in here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739840288


   Should we also respect `spark.sql.runSQLOnFiles` in Spark?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537761839



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);

Review comment:
       agreed. Lazy mistake.  Fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r533647251



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -165,6 +179,7 @@ public StagedTable stageCreate(Identifier ident, StructType schema, Transform[]
                                  Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
     try {
+      // can't stage a hadoop table

Review comment:
       I think this is okay, but if staging isn't supported then a path identifier should fail instead of going ahead without using the path. Otherwise, this will attempt to create a table in the Iceberg catalog with a crazy name, which would probably fail.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739012585


   I think we should keep the scope of this small. There isn't a need to address how to embed path-based tables in SQL commands right now. However we choose to do that would probably be a different solution. Right now, we need to unblock multicatalog support in `IcebergSource` by having a way to pass the path through as a table identifier, and the `PathIdentifier` option works well for that.
   
   For the larger question about `ALTER`, `DROP`, etc., I think the short-term solution is to define a HadoopCatalog that can handle those SQL commands.
   
   In the long term, I've advocated that Spark should have some way of identifying and passing path identifiers to plugins. The problem here is that no one seems to know what the behavior or path-based tables in SQL is or should be. But I think that a similar `PathIdentifier` approach would work once that is addressed in the upstream community.
   
   Last, if we want to support identifiers with a catalog and a quoted path, we can do that any time by choosing when to interpret the catalog name as a quoted path.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-741128713


   Thanks for the fixes! I merged this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537746849



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +193,162 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties) {
+    return buildTable(location, schema).withPartitionSpec(spec).withProperties(properties).createTransaction();
+  }
+
+  /**
+   * Start a transaction to replace a table.
+   *
+   * @param location a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @param orCreate whether to create the table if not exists
+   * @return a {@link Transaction} to replace the table
+   * @throws NoSuchTableException if the table doesn't exist and orCreate is false
+   */
+  public Transaction newReplaceTableTransaction(
+      String location,
+      Schema schema,
+      PartitionSpec spec,
+      Map<String, String> properties,
+      boolean orCreate) {
+
+
+    Catalog.TableBuilder builder = buildTable(location, schema).withPartitionSpec(spec).withProperties(properties);
+    return orCreate ? builder.createOrReplaceTransaction() : builder.replaceTransaction();
+  }
+
+  public Catalog.TableBuilder buildTable(String location, Schema schema) {
+    return new HadoopTableBuilder(location, schema);
+  }
+
+  private class HadoopTableBuilder implements Catalog.TableBuilder {
+    private final String location;
+    private final Schema schema;
+    private final ImmutableMap.Builder<String, String> propertiesBuilder = ImmutableMap.builder();
+    private PartitionSpec spec = PartitionSpec.unpartitioned();
+    private SortOrder sortOrder = SortOrder.unsorted();
+
+
+    HadoopTableBuilder(String location, Schema schema) {
+      this.location = location;
+      this.schema = schema;
+    }
+
+    @Override
+    public Catalog.TableBuilder withPartitionSpec(PartitionSpec newSpec) {
+      this.spec = newSpec != null ? newSpec : PartitionSpec.unpartitioned();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withSortOrder(SortOrder newSortOrder) {
+      this.sortOrder = newSortOrder != null ? newSortOrder : SortOrder.unsorted();
+      return this;
+    }
+
+    @Override
+    public Catalog.TableBuilder withLocation(String newLocation) {
+      return this;

Review comment:
       Agreed. Have added a precondition check.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537759985



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ?
+        new String[]{Joiner.on("/").join(pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]))} :

Review comment:
       There should be no need to call `toArray` here because the joiner accepts `Iterable<?>`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536373279



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -209,4 +278,35 @@ public void setConf(Configuration conf) {
   public Configuration getConf() {
     return conf;
   }
+
+
+  /**
+   * Check to see if the location is a potential Hadoop table by checking if its an absolute path on some filesystem.
+   */
+  public static boolean isHadoopTable(String location) {

Review comment:
       How about `isValidLocation`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740638215


   I have added a simple test to ensure the correct behaviour of the `SparkCatalog` when passed a `PathIdentifier`. It smoked out one minor bug so thanks @RussellSpitzer!
   
   All is up to date now and I think we are just about ready!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537766691



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -453,4 +467,21 @@ private static void commitChanges(Table table, SetProperty setLocation, SetPrope
 
     transaction.commitTransaction();
   }
+
+  private static boolean isPathIdentifier(Identifier ident) {
+    return ident instanceof PathIdentifier;
+  }
+
+  private static void checkNotPathIdentifier(Identifier identifier, String method) {
+    if (identifier instanceof PathIdentifier) {
+      throw new IllegalArgumentException(String.format("Cannot pass path based identifier to %s method. %s is a path.",
+          method, identifier));
+    }
+  }
+
+  private <T> T pathOrTable(Identifier ident, Function<String, T> path, Function<TableIdentifier, T> table) {

Review comment:
       Looks like there are 3 ways this is called:
   1. To load a table
   2. To create a builder
   3. To drop a table
   
   I think it would be a bit cleaner to make the first 2 into private methods (`load` and `newBuilder`) rather than using `pathOrTable` with the same functions in lots of places.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734932589


   > > I was thinking about this a bit differently, rather than providing a full catalog I was thinking that we just have the various spark catalogs treat file types as namespaces. Then we just have them switch to returning Hadoop tables in the load table method.
   > 
   > I was thinking this way originally as well. My first attempt at it made for weird interactions between `BaseMetastoreCatalog` and the individual catalogs. And a bunch of dupe code in the various `Catalog` impls. I am still not satisfied w/ this solution as the hadoop delegate catalog is a bit strange but at least it centralises the logic a bit.
   
   I re-read what you said earlier @RussellSpitzer and I realized I mis-read it. Your suggestion to just modify the spark catalogs may be easier but it still has a lot of duplication and would require a mechanism to handle `StagedTable`s with `HadoopTables`. I believe the change will end up very similar to my current patch. I can still do the change if you want to see the difference?
   
   *Note* regardless of how we implement the change `DROP` still causes a problem in session catalogs, drop never passes through our session catalog.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537758049



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ? pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]) :
+        new String[0];

Review comment:
       > Shouldn't namespace be just iceberg and name location to match built-in sources?
   
   I don't think so. Those identifiers would not be passed to Iceberg as `PathIdentifier` so there isn't a strong reason to make Iceberg's `PathIdentifier` use the same convention. They are all "iceberg" identifiers.
   
   > To me namespace + name should equal location.
   
   I'm fine either way. I do think that it makes sense to omit it from the namespace (["s3://bucket/path/to"] and "table"). The main requirement is that the location should be available unmodified when this is used.
   
   Sounds like we agree that the table name should be the last directory name to make the [subquery alias](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L972) work.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-736584986


   I have added `ALTER` tests and have skipped `DROP` when using SessionCatalog. I have also used @RussellSpitzer's suggestion for implementation. Its a lot cleaner and smaller but means we cant perform certain operations on filesystem tables: some transactions, renames and list by namespace.
   
   I am not a huge fan of identifying a path based table by simply checking if there is a `/` in the name. Any thoughts on how to make this less fragile?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r534171540



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -165,6 +179,7 @@ public StagedTable stageCreate(Identifier ident, StructType schema, Transform[]
                                  Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
     try {
+      // can't stage a hadoop table

Review comment:
       > I'd prefer to create a builder that can construct the transactions for path tables. We could add it to HadoopTables.
   
   Agreed. Added in most recent update




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740127934


   Looks good overall. I think I'd prefer to refactor the load and builder creation a little bit, and add that check to the Hadoop builder. Thanks @rymurr!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536930585



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -147,13 +156,23 @@ public SparkTable createTable(Identifier ident, StructType schema,
                                 Transform[] transforms,
                                 Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
+    TableIdentifier tableIdentifier = buildIdentifier(ident);
     try {
-      Table icebergTable = icebergCatalog.createTable(
-          buildIdentifier(ident),
-          icebergSchema,
-          Spark3Util.toPartitionSpec(icebergSchema, transforms),
-          properties.get("location"),
-          Spark3Util.rebuildCreateProperties(properties));
+      Table icebergTable;
+      if (isPathIdentifier(ident)) {

Review comment:
       Opened https://github.com/apache/iceberg/pull/1879 with this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-737756408


   Sorry I think I am pretty late in the whole conversation, completely forgot to follow up on #1783 after my initial comment, still trying to catch up...
   
   I also like the idea of a custom `PathIdentifier`, but it seems to only work for loading an existing table through `SupportsCatalogOptions`, and as @rymurr says I don't think it handles those `TableCatalog` operations well. The Spark plan would directly use `Identifier.of(...)` which uses its default `IdentifierImpl`.
   
   For these DDL, I actually like the approach that Anton mentioned in #1306 about wrapping `HadoopTables` into a catalog, for example `HadoopPathCatalog`, so that we can run something like `CREATE TABLE catalog_name."path/to/table" USING iceberg`. I don't think using `LOCATION` keyword is an option for us, not because of the dummy table name, but because not all ALTER and DELETE statements support it natively in open source Spark. That proposal was the last conversation of the issue and never got follow up, was there any concern with that approach not mentioned there?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537388883



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -164,13 +183,24 @@ public SparkTable createTable(Identifier ident, StructType schema,
   public StagedTable stageCreate(Identifier ident, StructType schema, Transform[] transforms,
                                  Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
+    TableIdentifier tableIdentifier = buildIdentifier(ident);
     try {
-      return new StagedSparkTable(icebergCatalog.newCreateTableTransaction(
-          buildIdentifier(ident),
-          icebergSchema,
-          Spark3Util.toPartitionSpec(icebergSchema, transforms),
-          properties.get("location"),
-          Spark3Util.rebuildCreateProperties(properties)));
+      Transaction transaction;
+      if (isPathIdentifier(ident)) {
+        transaction = tables.newCreateTableTransaction(
+            tableIdentifier.name(),

Review comment:
       yes, fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734828999


   This is still a bit rough, doesn't pass 1 test and I haven't tested `ALTER` commands but I wanted to get it out for review early. 
   
   One key issue:
   It appears that deleting tables is only a `V1` Datasource operation from the Session Catalog currently in Spark (see [here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala#L366)). So when trying to drop a table via file path eg: ```DROP TABLE `file://path/to/iceberg/table\` ``` when using a Session catalog you get an error. This is because the V1 Session Catalog looks in Hive and doesn't find the table. 
   
   The `DROP` commands work when using a `HiveCatalog` and the Session Catalog because the table was created via a `HiveCatalog` and our `SparkSessionCatalog` and deleted via a V1 Session Catalog but both are talking to Hive so we don't notice.
   
   In my mind this means the Spark3 tests that run against the session catalog are (partially) broken as the `DROP` command won't work for anything other than a Hive based table.
   
   So:
   * should we document this limitation and ignore?
   * should we fix it on the Spark side?
   * Should we fix it on our side (I honestly don't know how to though)?
   
   @rdblue @aokolnychyi any thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536930840



##########
File path: core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java
##########
@@ -200,6 +197,76 @@ TableOperations newTableOps(String location) {
     }
   }
 
+  private TableMetadata tableMetadata(Schema schema, PartitionSpec spec, SortOrder order,
+                                      Map<String, String> properties, String location) {
+    Preconditions.checkNotNull(schema, "A table schema is required");
+
+    Map<String, String> tableProps = properties == null ? ImmutableMap.of() : properties;
+    PartitionSpec partitionSpec = spec == null ? PartitionSpec.unpartitioned() : spec;
+    SortOrder sortOrder = order == null ? SortOrder.unsorted() : order;
+    return TableMetadata.newTableMetadata(schema, partitionSpec, sortOrder, location, tableProps);
+  }
+
+  /**
+   * Start a transaction to create a table.
+   *
+   * @param identifier a location for the table
+   * @param schema a schema
+   * @param spec a partition spec
+   * @param properties a string map of table properties
+   * @return a {@link Transaction} to create the table
+   * @throws AlreadyExistsException if the table already exists
+   */
+  public Transaction newCreateTableTransaction(

Review comment:
       Can this use `TableBuilder` instead? That would make the `SparkCatalog` implementation a lot cleaner because the code to configure the builder could be shared.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: WIP Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-734869844


   
   > Here is an example @aokolnychyi was typing up on the create ticket
   > 
   > ```
   > // Case 2.1: migrate a location using HadoopTables (or path-based catalog) for target
   > 
   > MIGRATE TABLE parquet.`path/to/table`
   > USING iceberg
   > TBLPROPERTIES (
   >   'key' 'value'
   > )
   > ```
   
   I like this example too. A very useful feature. This is slightly different/more complicated though right? That would add a sql extension and a separate impl to do write the metadata of the new table.  
   
   Or do you mean the path based table is already an iceberg table and the migrate is effectively reduced to a rename from `path` to `tablename` ?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536929426



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -147,13 +156,23 @@ public SparkTable createTable(Identifier ident, StructType schema,
                                 Transform[] transforms,
                                 Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
+    TableIdentifier tableIdentifier = buildIdentifier(ident);
     try {
-      Table icebergTable = icebergCatalog.createTable(
-          buildIdentifier(ident),
-          icebergSchema,
-          Spark3Util.toPartitionSpec(icebergSchema, transforms),
-          properties.get("location"),
-          Spark3Util.rebuildCreateProperties(properties));
+      Table icebergTable;
+      if (isPathIdentifier(ident)) {

Review comment:
       I think this would be cleaner using the builder. The only problem there is that the builder isn't implemented for `CachingCatalog` yet. I'll open a PR with that builder. Then you'd be able to have an if to create the builder, then configure the builder the same way for spec, properties, identifier, etc.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue closed pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue closed pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740102113


   > Should we also respect spark.sql.runSQLOnFiles in Spark? I am not sure whether we should match built-in sources here.
   
   I don't think so. That setting is for v1. Instead, Spark needs to define how it will pass path-based tables to catalogs, and what the behavior requirements are for those tables. Right now, no one has done the work to find out what the behavior of v1 is or to try to build consensus in the Spark community about what it should be. I think that means that Iceberg should avoid Spark's path-based syntax for now and focus in this PR on the narrower case of passing identifiers for paths used in `DataFrameReader`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r537763401



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/PathIdentifier.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.base.Splitter;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.spark.sql.connector.catalog.Identifier;
+
+public class PathIdentifier implements Identifier {
+  private final String[] namespace;
+  private final String location;
+  private final String name;
+
+  public PathIdentifier(String location) {
+    this.location = location;
+    List<String> pathParts = Splitter.on("/").splitToList(location);
+    name = Iterables.getLast(pathParts);
+    namespace = pathParts.size() > 1 ? pathParts.subList(0, pathParts.size() - 1).toArray(new String[0]) :
+        new String[0];

Review comment:
       cool, have left it as is. Thanks for the pointer to subquery alias!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536929513



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -257,6 +317,7 @@ public boolean dropTable(Identifier ident) {
   @Override
   public void renameTable(Identifier from, Identifier to) throws NoSuchTableException, TableAlreadyExistsException {
     try {
+      // can't rename hadoop tables

Review comment:
       This should throw an exception if it can't rename, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-738790926


   Thanks for the comment @jackye1995 , I have added a check to reject the path table if it has a namespace. I have also moved the checks to `HadoopTables`, good suggestion.
   
   Re path identifiers. Delta allows [absolute paths](https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala#L538) only so I don't think its outrageous if we limit paths to absolute paths only. I am inclined to use a check similar to what they do. Thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-737233889


   Just a bit of clarification on `PathIdentifier` and your above comment @rdblue: 
   
   You are suggesting that the change in #1783 returns a `PathIdentifier`  from the `SupportsCatalogOptions.extractIdentifier` method? I like that and it simplifies some of the logic in this PR. If I understand correctly in the case of stored procedures or other places where we have control we can use `PathIdentifier` or use the other mechanisms available to pass path information. Correct?
   
   What I am unsure about is how to handle path based `CREATE`, `DROP`, `ALTER` comamands. We don't have much control over the `Identifier` and we would still have to inspect if its a path. Unless we do a `CREAT TABLE x AS iceberg LOCATION path`, which works but the table name is redundant/ignored.  Or are you suggesting this PR doesn't handle path based DDL coming from Spark?
   
   FWIW I agree that we shouldn't have a 'special' namespace for importing path based tables, however my approach of ignoring namespaces if the name is a path probably isn't quite right either.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536931130



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -257,6 +317,7 @@ public boolean dropTable(Identifier ident) {
   @Override
   public void renameTable(Identifier from, Identifier to) throws NoSuchTableException, TableAlreadyExistsException {
     try {
+      // can't rename hadoop tables

Review comment:
       I think this should have a method like `checkNotPathIdentifier` that throws an IllegalArgumentException if a `PathIdentifier` is passed, so that we can ensure that they aren't passed to methods that don't support paths.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-740130491


   What are the plans for a test suite on this? I think this is all going in the right direction I just want to make sure we have a good set of checks to go along with it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536929669



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -453,4 +520,5 @@ private static void commitChanges(Table table, SetProperty setLocation, SetPrope
 
     transaction.commitTransaction();
   }
+

Review comment:
       Nit: unnecessary newline.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rymurr commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rymurr commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739839023


   Thanks for the review @rdblue I think its looking much better now! The build will fail until #1879 is merged. I will rebase then hopefully we will be ready to go :-)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-739013885


   > Delta allows absolute paths only . . .
   
   Considering the behavior for SQL is undefined, I would rather not add support for this right away. Delta needs this because it already supported these identifiers in v1 and has to be compatible. But I think we should have a use case that requires this before adding support.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#discussion_r536929426



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -147,13 +156,23 @@ public SparkTable createTable(Identifier ident, StructType schema,
                                 Transform[] transforms,
                                 Map<String, String> properties) throws TableAlreadyExistsException {
     Schema icebergSchema = SparkSchemaUtil.convert(schema);
+    TableIdentifier tableIdentifier = buildIdentifier(ident);
     try {
-      Table icebergTable = icebergCatalog.createTable(
-          buildIdentifier(ident),
-          icebergSchema,
-          Spark3Util.toPartitionSpec(icebergSchema, transforms),
-          properties.get("location"),
-          Spark3Util.rebuildCreateProperties(properties));
+      Table icebergTable;
+      if (isPathIdentifier(ident)) {

Review comment:
       I think this would be cleaner using the builder. The only problem there is that the builder isn't implemented for `CachingCatalog` yet. I'll open a PR with that builder. Then you'd be able to have an if to create the builder, then configure the builder the same way for spec, properties, identifier, etc.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #1843: Support for file paths in SparkCatalogs via HadoopTables

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1843:
URL: https://github.com/apache/iceberg/pull/1843#issuecomment-736603260


   I'm not sure we want to allow those other operations, since I'm not sure they actually have a lot of meaning. Like a rename on a path based table in a filesystem without renames means a full copy of the entire dataset. So it may be best to treat file based path's as static or at least immovable.
   
   I think we probably could identify path based tables based on the namespace too, like "parquet or file" or something. I'll take a look today


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org