You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hvanhovell (via GitHub)" <gi...@apache.org> on 2023/05/01 17:01:40 UTC

[GitHub] [spark] hvanhovell commented on a diff in pull request #40997: [SPARK-43321][Connect] Dataset#Joinwith

hvanhovell commented on code in PR #40997:
URL: https://github.com/apache/spark/pull/40997#discussion_r1181722000


##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -837,6 +837,76 @@ class Dataset[T] private[sql] (
     }
   }
 
+  /**
+   * Joins this Dataset returning a `Tuple2` for each pair where `condition` evaluates to true.
+   *
+   * This is similar to the relation `join` function with one important difference in the result
+   * schema. Since `joinWith` preserves objects present on either side of the join, the result
+   * schema is similarly nested into a tuple under the column names `_1` and `_2`.
+   *
+   * This type of join can be useful both for preserving type-safety with the original object
+   * types as well as working with relational data where either side of the join has column names
+   * in common.
+   *
+   * @param other
+   *   Right side of the join.
+   * @param condition
+   *   Join expression.
+   * @param joinType
+   *   Type of join to perform. Default `inner`. Must be one of: `inner`, `cross`, `outer`,
+   *   `full`, `fullouter`,`full_outer`, `left`, `leftouter`, `left_outer`, `right`, `rightouter`,
+   *   `right_outer`.
+   *
+   * @group typedrel
+   * @since 3.5.0
+   */
+  def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)] = {
+    val joinTypeValue = toJoinType(joinType, skipSemiAnti = true)
+    val joinedNullables = joinTypeValue match {
+      case proto.Join.JoinType.JOIN_TYPE_INNER | proto.Join.JoinType.JOIN_TYPE_CROSS =>
+        Seq(false, false)
+      case proto.Join.JoinType.JOIN_TYPE_FULL_OUTER =>
+        Seq(true, true)
+      case proto.Join.JoinType.JOIN_TYPE_LEFT_OUTER =>
+        Seq(false, true)
+      case proto.Join.JoinType.JOIN_TYPE_RIGHT_OUTER =>
+        Seq(true, false)
+      case e =>
+        throw new IllegalArgumentException(s"Unsupported join type `joinType`: $e")
+    }
+
+    val tupleEncoder = ProductEncoder
+      .tuple(Seq(this.encoder, other.encoder), Some(joinedNullables))
+      .asInstanceOf[AgnosticEncoder[(T, U)]]
+
+    sparkSession.newDataset(tupleEncoder) { builder =>
+      val joinBuilder = builder.getJoinBuilder
+      joinBuilder
+        .setLeft(plan.getRoot)
+        .setRight(other.plan.getRoot)
+        .setJoinType(joinTypeValue)
+        .setJoinCondition(condition.expr)
+        .setLeftSchema(DataTypeProtoConverter.toConnectProtoType(this.schema))

Review Comment:
   Is there a way to avoid the RPCs here? The input schema(s) can actually change between the schema call and the actual materialization of the plan. IMO it would be better to state in the proto we want the join to produce a different result shape, instead of explicitly sending the schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org