You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "giucris (via GitHub)" <gi...@apache.org> on 2023/05/14 06:33:44 UTC

[GitHub] [iceberg] giucris opened a new pull request, #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.3

giucris opened a new pull request, #7607:
URL: https://github.com/apache/iceberg/pull/7607

   Hi everyone. 
   Here a proposal to add on Iceberg an IcebergMergeInto dsl. 
   
   closes #3665 
   
   **What has been added**
   A scala/java MergeInto Iceberg API. 
   
   **What has been removed?**
   Nothing
   
   **What has benn changed?**
   * A test utility method to easily create Dataset, previously it was a private ones
   
   **What is missing**
   * Add to other spark versions
   * Complete other test cases
   * Docs enhancement
   
   **How it was developed**
   * A new scala class has been added inside spark extensions. A simple builder that generates and execute an UnresolvedMergeIntoIcebergTable. The same one that is intercepted and rewritten during the execution of standard sql queries. 
   
   **How it was tested**
   * The same TestMerge tests were duplicated for test merge api
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Added Scala/Java IcebergMergeInto Api on spark3.4 [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-2319518893

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Added Scala/Java IcebergMergeInto Api on spark3.4 [iceberg]

Posted by "dsomesh (via GitHub)" <gi...@apache.org>.
dsomesh commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r3020264603


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   @giucris did you conclude on this? I would like to pass write options in merge query which seems to be missing now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334860227


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateAction(
+          condition,
+          set.map(x => Assignment(expr(x._1).expr, x._2.expr)).toSeq))
+    }
+
+
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenNotMatched actions
+   */
+  class IcebergMergeWhenNotMatchedBuilder(
+                                           private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,

Review Comment:
   Hi @ConeyLiu this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334766142


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
+
+    createAndInitTable("id INT, dep STRING");
+    Dataset<Row> source = spark.createDataset(Collections.singletonList(1), Encoders.INT()).toDF();
+
+    sql(
+        "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
+        tableName, MERGE_ISOLATION_LEVEL, "snapshot");
+
+    sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
+    createBranchIfNeeded();
+
+    Table table = Spark3Util.loadIcebergTable(spark, tableName);
+
+    ExecutorService executorService =
+        MoreExecutors.getExitingExecutorService(
+            (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
+
+    AtomicInteger barrier = new AtomicInteger(0);
+    AtomicBoolean shouldAppend = new AtomicBoolean(true);
+
+    // merge thread
+    Future<?> mergeFuture =
+        executorService.submit(
+            () -> {
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (barrier.get() < numOperations * 2) {
+                  sleep(10);
+                }
+                IcebergMergeInto.apply(tableName)
+                    .using(source.as("source"))
+                    .when(format("%s.id = source.value", tableName))
+                    .whenMatched()
+                    .update(
+                        new HashMap<String, Column>() {
+                          {
+                            put("dep", lit("x"));
+                          }
+                        })
+                    .merge();
+                barrier.incrementAndGet();
+              }
+            });
+
+    // append thread
+    Future<?> appendFuture =
+        executorService.submit(
+            () -> {
+              GenericRecord record = GenericRecord.create(table.schema());
+              record.set(0, 1); // id
+              record.set(1, "hr"); // dep
+
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (shouldAppend.get() && barrier.get() < numOperations * 2) {
+                  sleep(10);
+                }
+
+                if (!shouldAppend.get()) {
+                  return;
+                }
+
+                for (int numAppends = 0; numAppends < 5; numAppends++) {
+                  DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
+                  table.newFastAppend().appendFile(dataFile).commit();
+                  sleep(10);
+                }
+
+                barrier.incrementAndGet();
+              }
+            });
+
+    try {
+      Assertions.assertThatThrownBy(mergeFuture::get)
+          .isInstanceOf(ExecutionException.class)
+          .cause()
+          .isInstanceOf(IllegalStateException.class)
+          .hasMessageContaining("the table has been concurrently modified");
+    } finally {
+      shouldAppend.set(false);
+      appendFuture.cancel(true);
+    }
+
+    executorService.shutdown();
+    Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));

Review Comment:
   Hi @RussellSpitzer this test is a copy of the existing test inside the SQLMerge test cases.
    
   What is your opinion on this? IMHO the ideal is to keep the test battery identical or remove some tests if not of interest.
   
   Changing a parameter between an existing test (in SQLMerge) could lead to red tests on one side and green tests on the other. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282430730


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
+
+    createAndInitTable("id INT, dep STRING");
+    Dataset<Row> source = spark.createDataset(Collections.singletonList(1), Encoders.INT()).toDF();
+
+    sql(
+        "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
+        tableName, MERGE_ISOLATION_LEVEL, "snapshot");
+
+    sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
+    createBranchIfNeeded();
+
+    Table table = Spark3Util.loadIcebergTable(spark, tableName);
+
+    ExecutorService executorService =
+        MoreExecutors.getExitingExecutorService(
+            (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
+
+    AtomicInteger barrier = new AtomicInteger(0);
+    AtomicBoolean shouldAppend = new AtomicBoolean(true);
+
+    // merge thread
+    Future<?> mergeFuture =
+        executorService.submit(
+            () -> {
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (barrier.get() < numOperations * 2) {

Review Comment:
   integer overflow here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223912025


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =

Review Comment:
   Why is the target table using the table name while the source table using a `Dataset`? Should we support the table name as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334841493


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   Hi @ConeyLiu  I think it should be possible. I will try to configure it



##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java:
##########
@@ -214,15 +214,15 @@ protected void createOrReplaceView(String name, String jsonData) {
   }
 
   protected void createOrReplaceView(String name, String schema, String jsonData) {
-    Dataset<Row> ds = toDS(schema, jsonData);
+    Dataset<Row> ds = createDataset(schema, jsonData);
     ds.createOrReplaceTempView(name);
   }
 
   protected <T> void createOrReplaceView(String name, List<T> data, Encoder<T> encoder) {
     spark.createDataset(data, encoder).createOrReplaceTempView(name);
   }
 
-  private Dataset<Row> toDS(String schema, String jsonData) {
+  protected Dataset<Row> createDataset(String schema, String jsonData) {

Review Comment:
   @ConeyLiu @RussellSpitzer this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334832891


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {

Review Comment:
   Hi @RussellSpitzer this test is a copy of the existing test inside the SQLMerge test cases.
   
   I have started the development with all battery of test that already exist in the SQLMerge. 
   
   If this test does not make any sense I can drop it. 
   But this is already in the standard sql merge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Added Scala/Java IcebergMergeInto Api on spark3.4 [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-2334958835

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334840774


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =

Review Comment:
   Hi @ConeyLiu and @RussellSpitzer this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1582607600

   Sorry. 
   I was a bit underpressure on last 3 weeks. 
   I will try to complete as soon I have some time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1584078304

   Thanks @giucris for the great work. left some API and code style comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Added Scala/Java IcebergMergeInto Api on spark3.4 [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4
URL: https://github.com/apache/iceberg/pull/7607


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223918322


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {

Review Comment:
   code tyle



##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateAction(
+          condition,
+          set.map(x => Assignment(expr(x._1).expr, x._2.expr)).toSeq))
+    }
+

Review Comment:
   blank line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Neuw84 commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "Neuw84 (via GitHub)" <gi...@apache.org>.
Neuw84 commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1582428184

   Hi, 
   
   It would be great if we can push this feature to merge. 
   
   I could help if needed too on the Java side. 
   
   Let me know! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334859682


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,

Review Comment:
   Hi @ConeyLiu it should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282430418


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {

Review Comment:
   Not sure what the goal of this test is? We aren't really testing the API here but the actual underlying implementation of the merge. The test could fit in the merge implementation tests, I also would imagine that we would want to use Tasks.foreach or something like that to do our concurrent tasks, should be a bit clearer than the current semaphore and multi threaded approach? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334841023


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+

Review Comment:
   @ConeyLiu this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223914128


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =

Review Comment:
   The `when` seems a little confused with `whenMatched` / `whenNotMatched`. What about `on`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Neuw84 commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "Neuw84 (via GitHub)" <gi...@apache.org>.
Neuw84 commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1582612694

   @giucris  let me know if I can help on anything 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1551813429

   Hi @RussellSpitzer 
   
   The API is inspired by the delta ones to avoid confusion in those switching from one table format to another. 
   
   I think a Java Class with a static method that under calls the same builder might be enough to solve the "apply" problem and make everything look more "java" oriented. 
   
   I will try to add it
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1582586509

   Other reviewers would be welcome, I'm a bit pressed for time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282420262


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =

Review Comment:
   I would keep this as "on" because fundamentally this is setting shuffle join conditions (join on) while the other functions are only exercised at execution time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223917612


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]

Review Comment:
   Maybe add doc for it



##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]

Review Comment:
   same here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1551780147

   First high level comment, I think we really need an underlying Java API as well. I think there should be a first class static class in java that can be used similar to the extension here. Other than that I think the API looks good at first glance, I assume it is matching the Delta api?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334837859


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =

Review Comment:
   Hi @ConeyLiu can we consider this thread as solved? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334958313


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   I did some research and I need more details on how you would like the merge api options to work. 
   
   1. Do we want to set the options at the individual action level or at the global merge into level or both? 
   
   2. What kind of options do we want to support?  Runtime write config or Table properties or both?
   
   To date and following the iceberg docs, the options really useful (IMHO) for merge, e.g., write.merge.mode, write.delete.mode, etc., are table properties and are currently set on the target table side during create or after altering the table. 
   
   However, there are some runtime configurations that could be provided at write time, [these](https://iceberg.apache.org/docs/latest/spark-configuration/#write-options)
   
   3. In case we also want to support table properties, what is expected after the merge operation with configured table properties? Target table containing the new properties or we want that the properties are used only in that runtime (In this case I don't know how feasible it is, I need to investigate further)
   
   
   Given all these open points, can we addressing this need as a new feature, thus in a subsequent PR of the IcebergMergeInto API?  
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1732297343

   Hi @RussellSpitzer and @ConeyLiu. 
   
   First of all, thank you for your time and for your review and excuse me for coming back after a while. I've been a little busy. 
   
   I have been trying to fix all your threads. On others I opened a discussion because I need more clarity on how to proceed. 
   
   Now there is no need to have a Java API since I have included a table method as the main method to start the builder to avoid duplicate code. 
   
   ```scala
   IcebergMergeInto
      .table("icebergTable")
     .using(ds.as("source"))
     .on("source.id == icebergTable.id")
     .whenMatched()
     .updateAll()
     .whenNotMatched()
     .insertAll()
     .merge()
   ```
   
   Let me know what you think, which threads can be closed, which ones to keep ongoing and which ones address into another PR.
   
   Thank you again :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1662998068

   Took a look today, as noted by @ConeyLiu we have a lot of code style issues to clean up here. The tests are also a little confusing because I think some of them are covering overall merge behavior and some cover the api. Let's try to minimize the surface area of this PR if we can.
   
   The API itself looks ok to me, I left a few comments there. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223915981


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+

Review Comment:
   blank line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223916115


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =

Review Comment:
   @ConeyLiu In my opinion the main use of a Merge api is to handle the merge between a source dataset and a target table. To avoid having to register the dataset as a temporary table and perform the operation in sql. 
   
   In case both the "delta" and the target table are already registered I don't see why use this api in favour of a simple SQL query. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223917927


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]

Review Comment:
   same as here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223920594


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   Is there possible to provide the option functions to set the configs for the merge actions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223920811


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java:
##########
@@ -214,15 +214,15 @@ protected void createOrReplaceView(String name, String jsonData) {
   }
 
   protected void createOrReplaceView(String name, String schema, String jsonData) {
-    Dataset<Row> ds = toDS(schema, jsonData);
+    Dataset<Row> ds = createDataset(schema, jsonData);
     ds.createOrReplaceTempView(name);
   }
 
   protected <T> void createOrReplaceView(String name, List<T> data, Encoder<T> encoder) {
     spark.createDataset(data, encoder).createOrReplaceTempView(name);
   }
 
-  private Dataset<Row> toDS(String schema, String jsonData) {
+  protected Dataset<Row> createDataset(String schema, String jsonData) {

Review Comment:
   why change the name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223929943


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   @ConeyLiu I think it's doable, I don't think it's something complex to add. Let me try.



##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java:
##########
@@ -214,15 +214,15 @@ protected void createOrReplaceView(String name, String jsonData) {
   }
 
   protected void createOrReplaceView(String name, String schema, String jsonData) {
-    Dataset<Row> ds = toDS(schema, jsonData);
+    Dataset<Row> ds = createDataset(schema, jsonData);
     ds.createOrReplaceTempView(name);
   }
 
   protected <T> void createOrReplaceView(String name, List<T> data, Encoder<T> encoder) {
     spark.createDataset(data, encoder).createOrReplaceTempView(name);
   }
 
-  private Dataset<Row> toDS(String schema, String jsonData) {
+  protected Dataset<Row> createDataset(String schema, String jsonData) {

Review Comment:
   @ConeyLiu Previously this method was private. and the name toDS could fit. Now it has been aligned with the other utility methods that are protected and all this method begin with 'create' prefix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282439133


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
+
+    createAndInitTable("id INT, dep STRING");
+    Dataset<Row> source = spark.createDataset(Collections.singletonList(1), Encoders.INT()).toDF();
+
+    sql(
+        "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
+        tableName, MERGE_ISOLATION_LEVEL, "snapshot");
+
+    sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
+    createBranchIfNeeded();
+
+    Table table = Spark3Util.loadIcebergTable(spark, tableName);
+
+    ExecutorService executorService =
+        MoreExecutors.getExitingExecutorService(
+            (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
+
+    AtomicInteger barrier = new AtomicInteger(0);
+    AtomicBoolean shouldAppend = new AtomicBoolean(true);
+
+    // merge thread
+    Future<?> mergeFuture =
+        executorService.submit(
+            () -> {
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (barrier.get() < numOperations * 2) {
+                  sleep(10);
+                }
+                IcebergMergeInto.apply(tableName)
+                    .using(source.as("source"))
+                    .when(format("%s.id = source.value", tableName))
+                    .whenMatched()
+                    .update(
+                        new HashMap<String, Column>() {
+                          {
+                            put("dep", lit("x"));
+                          }
+                        })
+                    .merge();
+                barrier.incrementAndGet();
+              }
+            });
+
+    // append thread
+    Future<?> appendFuture =
+        executorService.submit(
+            () -> {
+              GenericRecord record = GenericRecord.create(table.schema());
+              record.set(0, 1); // id
+              record.set(1, "hr"); // dep
+
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (shouldAppend.get() && barrier.get() < numOperations * 2) {
+                  sleep(10);
+                }
+
+                if (!shouldAppend.get()) {
+                  return;
+                }
+
+                for (int numAppends = 0; numAppends < 5; numAppends++) {
+                  DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
+                  table.newFastAppend().appendFile(dataFile).commit();
+                  sleep(10);
+                }
+
+                barrier.incrementAndGet();
+              }
+            });
+
+    try {
+      Assertions.assertThatThrownBy(mergeFuture::get)
+          .isInstanceOf(ExecutionException.class)
+          .cause()
+          .isInstanceOf(IllegalStateException.class)
+          .hasMessageContaining("the table has been concurrently modified");
+    } finally {
+      shouldAppend.set(false);
+      appendFuture.cancel(true);
+    }
+
+    executorService.shutdown();
+    Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));

Review Comment:
   we probably don't want to add in 2 minute tests if we can help it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282731838


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));

Review Comment:
   Hi @RussellSpitzer. All the tests added are a "brutal" copy and paste of the tests inside the "sql" merge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223916675


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,

Review Comment:
   code style



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223919545


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.

Review Comment:
   Should move this to the above object?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#issuecomment-1584091761

   > @giucris let me know if I can help on anything 👍
   
   @Neuw84 Feel free to contribute as you can


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334768396


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
+
+    createAndInitTable("id INT, dep STRING");
+    Dataset<Row> source = spark.createDataset(Collections.singletonList(1), Encoders.INT()).toDF();
+
+    sql(
+        "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
+        tableName, MERGE_ISOLATION_LEVEL, "snapshot");
+
+    sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
+    createBranchIfNeeded();
+
+    Table table = Spark3Util.loadIcebergTable(spark, tableName);
+
+    ExecutorService executorService =
+        MoreExecutors.getExitingExecutorService(
+            (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
+
+    AtomicInteger barrier = new AtomicInteger(0);
+    AtomicBoolean shouldAppend = new AtomicBoolean(true);
+
+    // merge thread
+    Future<?> mergeFuture =
+        executorService.submit(
+            () -> {
+              for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
+                while (barrier.get() < numOperations * 2) {

Review Comment:
   Hi @RussellSpitzer this test is a copy of the existing test inside the SQLMerge test cases.
   
   I could drop it or just fix the test, but I would have to first get a good understanding of what is being tested, but it should also be fixed on the SQLMerge test cases side, perhaps in a later MR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334841292


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.

Review Comment:
   @ConeyLiu this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223912392


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,

Review Comment:
   code style should be fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223918581


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateAction(
+          condition,
+          set.map(x => Assignment(expr(x._1).expr, x._2.expr)).toSeq))
+    }
+
+
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenNotMatched actions
+   */
+  class IcebergMergeWhenNotMatchedBuilder(
+                                           private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,

Review Comment:
   codestyle



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223918130


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]

Review Comment:
   same as here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1223917550


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =

Review Comment:
   @ConeyLiu I agree. 
   I use when to have similarity with Delta MergeApi. But I completely agree that on could be better then when.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282427268


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMergeAPI.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static java.lang.String.format;
+import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
+import static org.apache.spark.sql.functions.lit;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.iceberg.spark.Spark3Util;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.SnapshotUtil;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+
+public class TestCopyOnWriteMergeAPI extends TestMergeAPI {
+
+  public TestCopyOnWriteMergeAPI(
+      String catalogName,
+      String implementation,
+      Map<String, String> config,
+      String fileFormat,
+      boolean vectorized,
+      String distributionMode,
+      String branch) {
+    super(catalogName, implementation, config, fileFormat, vectorized, distributionMode, branch);
+  }
+
+  @Override
+  protected Map<String, String> extraTableProperties() {
+    return ImmutableMap.of(
+        TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
+  }
+
+  @Test
+  public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
+    // this test can only be run with Hive tables as it requires a reliable lock
+    // also, the table cache must be enabled so that the same table instance can be reused
+    Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));

Review Comment:
   Not sure I understand why this can't be run with hadoop catalog. Should still be fine. The filesystem here should be the local fs and should be ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282418353


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {

Review Comment:
   This would be great, although I think we can't do that without a change of the actual Merge right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1282425375


##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java:
##########
@@ -214,15 +214,15 @@ protected void createOrReplaceView(String name, String jsonData) {
   }
 
   protected void createOrReplaceView(String name, String schema, String jsonData) {
-    Dataset<Row> ds = toDS(schema, jsonData);
+    Dataset<Row> ds = createDataset(schema, jsonData);
     ds.createOrReplaceTempView(name);
   }
 
   protected <T> void createOrReplaceView(String name, List<T> data, Encoder<T> encoder) {
     spark.createDataset(data, encoder).createOrReplaceTempView(name);
   }
 
-  private Dataset<Row> toDS(String schema, String jsonData) {
+  protected Dataset<Row> createDataset(String schema, String jsonData) {

Review Comment:
   I would agree that we shouldn't change method names if we can help it in a pr that is already quite large. I would just keep it as createDataset and do the change to toDS in another pr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334859950


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,

Review Comment:
   Hi @ConeyLiu it should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] giucris commented on a diff in pull request #7607: Spark: Added Scala/Java IcebergMergeInto Api on spark3.4

Posted by "giucris (via GitHub)" <gi...@apache.org>.
giucris commented on code in PR #7607:
URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334841176


##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {

Review Comment:
   @ConeyLiu this should be fixed now



##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark.extensions
+
+import org.apache.spark.sql.Column
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.functions.expr
+import scala.collection.JavaConverters
+
+
+object IcebergMergeInto {
+  /**
+   * Initialize an [[IcebergMergeIntoBuilder]].
+   *
+   * @param table : Target table name of the merge action
+   * @return [[IcebergMergeIntoBuilder]]
+   */
+  def apply(table: String): IcebergMergeIntoBuilder =
+    new IcebergMergeIntoBuilder(table, None, None, Seq.empty[MergeAction], Seq.empty[MergeAction])
+
+  /**
+   * Builder to specify an IcebergMergeInto action.
+   *
+   * It could be possible to provide any number of `whenMatched`
+   * and `whenNotMatched` actions.
+   *
+   *
+   * Scala Examples:
+   * {{{
+   *   val ds = ...
+   *   IcebergMergeInto("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   * JavaExamples:
+   * {{{
+   *   Dataset<Row> ds = ...
+   *   IcebergMergeInto.apply("icebergTable")
+   *      .using(ds.as("source")
+   *      .when("source.id = icebergTable.id")
+   *      .whenMatched("source.op = U")
+   *      .updateAll()
+   *      .whenMatched("source.op = D)
+   *      .delete()
+   *      .whenNotMatched()
+   *      .insertAll()
+   * }}}
+   *
+   */
+  class IcebergMergeIntoBuilder(
+                                 private val targetTable: String,
+                                 private val source: Option[Dataset[Row]],
+                                 private val whenCondition: Option[Expression],
+                                 private val whenMatchedActions: Seq[MergeAction],
+                                 private val whenNotMatchedActions: Seq[MergeAction]) {
+
+    /**
+     * Set the source dataset used during merge actions.
+     *
+     * @param source  : Dataset[Row]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def using(source: Dataset[Row]): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        targetTable,
+        Some(source),
+        whenCondition,
+        whenMatchedActions,
+        whenNotMatchedActions
+      )
+
+    /**
+     * Set the `when` condition of the merge action from a [[Column]]
+     *
+     * @param condition : [[Column]] expression
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: Column): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(targetTable, source, Some(condition.expr), whenMatchedActions, whenNotMatchedActions)
+
+    /**
+     * Set the `when` condition of the merge action from a [[String]] expression.
+     *
+     * @param condition : String
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def when(condition: String): IcebergMergeIntoBuilder =
+      when(expr(condition))
+
+    /**
+     * Set a `whenMatched` action.
+     *
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[Column]]
+     * @return IcebergMergeWhenMatchedBuilder
+     */
+    def whenMatched(condition: Column): IcebergMergeWhenMatchedBuilder =
+      new IcebergMergeWhenMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenMatched` action.
+     *
+     * @param condition : [[String]]
+     * @return [[IcebergMergeWhenMatchedBuilder]]
+     */
+    def whenMatched(condition: String): IcebergMergeWhenMatchedBuilder =
+      whenMatched(expr(condition))
+
+    /**
+     * Set a `whenNotMatched` action.
+     *
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, None)
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: Column): IcebergMergeWhenNotMatchedBuilder =
+      new IcebergMergeWhenNotMatchedBuilder(this, Some(condition.expr))
+
+    /**
+     * Set a conditionally `whenNotMatched` action
+     *
+     * @param condition : [[Column]]
+     * @return [[IcebergMergeWhenNotMatchedBuilder]]
+     */
+    def whenNotMatched(condition: String): IcebergMergeWhenNotMatchedBuilder =
+      whenNotMatched(expr(condition))
+
+    /**
+     * Internal method to append a NotMatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withNotMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions,
+        this.whenNotMatchedActions :+ mergeAction
+      )
+
+    /**
+     * Internal method to append a MatchedExpressions
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    private[iceberg] def withMatchedAction(mergeAction: MergeAction): IcebergMergeIntoBuilder =
+      new IcebergMergeIntoBuilder(
+        this.targetTable,
+        this.source,
+        this.whenCondition,
+        this.whenMatchedActions :+ mergeAction,
+        this.whenNotMatchedActions
+      )
+
+
+    /**
+     * Execute the merge action
+     */
+    def merge(): Unit = {
+      val mergeSourceDs = source
+        .getOrElse(throw new IllegalArgumentException("Merge statement require source Dataset"))
+
+      val mergeWhenCondition = whenCondition
+        .getOrElse(throw new IllegalArgumentException("Merge statement require whenCondition"))
+
+      val sparkSession = mergeSourceDs.sparkSession
+      val mergePlan = UnresolvedMergeIntoIcebergTable(
+        UnresolvedRelation(sparkSession.sessionState.sqlParser.parseMultipartIdentifier(targetTable)),
+        sparkSession.sessionState.analyzer.ResolveRelations(mergeSourceDs.queryExecution.logical),
+        MergeIntoContext(mergeWhenCondition, whenMatchedActions, whenNotMatchedActions)
+      )
+      runCommand(sparkSession, mergePlan)
+    }
+
+    private def runCommand(sparkSession: SparkSession, plan: LogicalPlan): Unit = {
+      val qe = sparkSession.sessionState.executePlan(plan)
+      qe.assertCommandExecuted()
+    }
+  }
+
+  /**
+   * Builder to specify IcebergMergeWhenMatched actions
+   *
+   */
+  class IcebergMergeWhenMatchedBuilder(
+                                        private val icebergMergeIntoBuilder: IcebergMergeIntoBuilder,
+                                        private val condition: Option[Expression]) {
+
+    /**
+     * Set an updateAll action.
+     * It will update the target records with all the column on the source dataset.
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateAll(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateStarAction(condition)
+      )
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(set)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def update(set: java.util.Map[String, Column]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(set.mapValues(expr).toMap)
+    }
+
+    /**
+     * Set an update action with a map of references between source and target table.
+     *
+     * It will update the target records accordingly with the assignment map provided as input.
+     *
+     * @param set : java.util.Map[String,String]
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def updateExpr(set: java.util.Map[String, String]): IcebergMergeIntoBuilder = {
+      updateAction(JavaConverters.mapAsScalaMap(set).mapValues(expr).toMap)
+    }
+
+    /**
+     * Set a delete action.
+     * It will delete the target
+     *
+     * @return [[IcebergMergeIntoBuilder]]
+     */
+    def delete(): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        DeleteAction(condition)
+      )
+    }
+
+    private def updateAction(set:Map[String,Column]): IcebergMergeIntoBuilder = {
+      icebergMergeIntoBuilder.withMatchedAction(
+        UpdateAction(
+          condition,
+          set.map(x => Assignment(expr(x._1).expr, x._2.expr)).toSeq))
+    }
+

Review Comment:
   @ConeyLiu this should be fixed now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org