You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/25 20:23:34 UTC
[GitHub] [spark] brandondahler opened a new pull request, #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
brandondahler opened a new pull request, #36343:
URL: https://github.com/apache/spark/pull/36343
### What changes were proposed in this pull request?
Adds 3 new syntactic sugar overloads to Dataset's join method as proposed in [SPARK-35739](https://issues.apache.org/jira/browse/SPARK-35739).
### Why are the changes needed?
Improved development experience for developers using Spark SQL, specifically when coding in Java.
Prior to changes the Seq overloads required developers to use less-known Java-to-Scala converter methods that made code less readable. The overloads internalize those converter calls for two of the new methods and the third method adds a single-item overload that is useful for both Java and Scala.
### Does this PR introduce _any_ user-facing change?
Yes, the three new overloads technically constitute an API change to the Dataset class. These overloads are net-new and have been commented appropriately in line with the existing methods.
### How was this patch tested?
Test cases were not added because it is unclear to me where/how syntactic sugar overloads fit into the testing suites (if at all). Happy to add them if I can be pointed in the correct direction.
* Changes were tested in Scala via spark-shell.
* Changes were tested in Java by modifying an example:
```
diff --git a/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java b/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java
index 86a9045d8a..342810c1e6 100644
--- a/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java
+++ b/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java
@@ -124,6 +124,10 @@ public class JavaSparkSQLExample {
// |-- age: long (nullable = true)
// |-- name: string (nullable = true)
+ df.join(df, new String[] {"age"}).show();
+ df.join(df, "age", "left").show();
+ df.join(df, new String[] {"age"}, "left").show();
+
// Select only the "name" column
df.select("name").show();
// +-------+
```
#### Notes
Re-opening of #33323 and #34923 with comments addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1111625226
Oh, @brandondahler mind enabling GitHub Actions in your forked repository? Apache Spark leverages the resources from the forked repository to run GitHub Actions build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1110423275
@brandondahler mind rebasing please? Seems like the test did not get triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1112756463
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
URL: https://github.com/apache/spark/pull/36343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1109668804
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a diff in pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #36343:
URL: https://github.com/apache/spark/pull/36343#discussion_r858018794
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -946,7 +946,21 @@ class Dataset[T] private[sql](
}
/**
- * Inner equi-join with another `DataFrame` using the given columns.
+ * (Java-specific) Inner equi-join with another `DataFrame` using the given columns. See the
+ * Scala-specific overload for more details.
+ *
+ * @param right Right side of the join operation.
+ * @param usingColumns Names of the columns to join on. This columns must exist on both sides.
+ *
+ * @group untypedrel
+ * @since 3.3.0
Review Comment:
(Make it 3.4.0 now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on a diff in pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
brandondahler commented on code in PR #36343:
URL: https://github.com/apache/spark/pull/36343#discussion_r858180941
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -946,7 +946,21 @@ class Dataset[T] private[sql](
}
/**
- * Inner equi-join with another `DataFrame` using the given columns.
+ * (Java-specific) Inner equi-join with another `DataFrame` using the given columns. See the
+ * Scala-specific overload for more details.
+ *
+ * @param right Right side of the join operation.
+ * @param usingColumns Names of the columns to join on. This columns must exist on both sides.
+ *
+ * @group untypedrel
+ * @since 3.4.0
+ */
+ def 4oin(right: Dataset[_], usingColumns: Array[String]): DataFrame = {
Review Comment:
Oh man that's embarrassing, I blame vim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1111145717
Rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1109214389
Bumped since to 3.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1109006814
Requesting review from previous reviewers from #33323 - @srowen and @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36343:
URL: https://github.com/apache/spark/pull/36343#discussion_r858180062
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -946,7 +946,21 @@ class Dataset[T] private[sql](
}
/**
- * Inner equi-join with another `DataFrame` using the given columns.
+ * (Java-specific) Inner equi-join with another `DataFrame` using the given columns. See the
+ * Scala-specific overload for more details.
+ *
+ * @param right Right side of the join operation.
+ * @param usingColumns Names of the columns to join on. This columns must exist on both sides.
+ *
+ * @group untypedrel
+ * @since 3.4.0
+ */
+ def 4oin(right: Dataset[_], usingColumns: Array[String]): DataFrame = {
Review Comment:
hmm.... `4oin` -> `join` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #36343: [SPARK-35739][SQL] Add Java-compatible Dataset.join overloads
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #36343:
URL: https://github.com/apache/spark/pull/36343#issuecomment-1109233251
Now with 100% less errant typos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org