You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/16 14:17:41 UTC
[GitHub] [spark] kozakana opened a new pull request #30803: [SPARK-][SQL]Can't set option 'cross' in join method.
kozakana opened a new pull request #30803:
URL: https://github.com/apache/spark/pull/30803
### What changes were proposed in this pull request?
[The PySpark documentation](https://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.join) says "Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti."
However, I get the following error when I set the cross option.
```
scala> val df1 = spark.createDataFrame(Seq((1,"a"),(2,"b")))
df1: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
scala> val df2 = spark.createDataFrame(Seq((1,"A"),(2,"B"), (3, "C")))
df2: org.apache.spark.sql.DataFrame = [_1: int, _2: string]
scala> df1.join(right = df2, usingColumns = Seq("_1"), joinType = "cross").show()
java.lang.IllegalArgumentException: requirement failed: Unsupported using join type Cross
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.sql.catalyst.plans.UsingJoin.<init>(joinTypes.scala:106)
at org.apache.spark.sql.Dataset.join(Dataset.scala:1025)
... 53 elided
```
### Why are the changes needed?
The documentation says cross option can be set, but when I try to set it, I get an java.lang.IllegalArgumentException.
### Does this PR introduce _any_ user-facing change?
Accepting this PR fix will behave the same as the documentation.
### How was this patch tested?
There is already a test for [JoinTypes](https://github.com/apache/spark/blob/1b9fd67904671ea08526bfb7a97d694815d47665/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/JoinTypesTest.scala), but I can't find a test for the join option itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750499068
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133319/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747915663
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37590/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747997449
@HyukjinKwon
Thank you for confirming. I try add a test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750408173
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37912/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748003804
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132990/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-746361252
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750396005
@HyukjinKwon
Thanks for your advice. I added test of cross join in [org.apache.spark.sql.DataFrameJoinSuite](https://github.com/apache/spark/pull/30803/commits/690ef81da1b06c189201782fe4cb1c332dd74287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-33897][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750749214
@HyukjinKwon
I created [SPARK-33897](https://issues.apache.org/jira/browse/SPARK-33897) in JIRA and modify title of github issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747915663
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37590/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750491107
**[Test build #133319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133319/testReport)** for PR 30803 at commit [`690ef81`](https://github.com/apache/spark/commit/690ef81da1b06c189201782fe4cb1c332dd74287).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747884469
It was probably a mistake. Can we add a test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750408173
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37912/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748600086
@HyukjinKwon
`join` methods has
```
def join(right: Dataset[_], joinExprs: Column, joinType: String): DataFrame
def join(right: Dataset[_], joinExprs: Column): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String]): DataFrame
def join(right: Dataset[_], usingColumn: String): DataFrame
def join(right: Dataset[_]): DataFrame
```
[ref. org.apache.spark.sql.Dataset](https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/sql/Dataset.html)
`join` method always requires a column name. So it may have been implemented as `crossJoin`. And, I discovery [this comment](https://github.com/apache/spark/blob/8197ee3b15265d39f05f192934b7d7e661713eaa/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L993-L995).
https://github.com/apache/spark/blob/8197ee3b15265d39f05f192934b7d7e661713eaa/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L993-L995
I suggest remove the cross option from the join method and use only the `crossJoin` method. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750499068
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133319/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750404816
**[Test build #133319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133319/testReport)** for PR 30803 at commit [`690ef81`](https://github.com/apache/spark/commit/690ef81da1b06c189201782fe4cb1c332dd74287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750404816
**[Test build #133319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133319/testReport)** for PR 30803 at commit [`690ef81`](https://github.com/apache/spark/commit/690ef81da1b06c189201782fe4cb1c332dd74287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747866768
Okay seems we should fix. @kozakana can you add a test case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30803:
URL: https://github.com/apache/spark/pull/30803#discussion_r544760162
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala
##########
@@ -102,7 +102,7 @@ case class NaturalJoin(tpe: JoinType) extends JoinType {
}
case class UsingJoin(tpe: JoinType, usingColumns: Seq[String]) extends JoinType {
- require(Seq(Inner, LeftOuter, LeftSemi, RightOuter, FullOuter, LeftAnti).contains(tpe),
+ require(Seq(Inner, LeftOuter, LeftSemi, RightOuter, FullOuter, LeftAnti, Cross).contains(tpe),
Review comment:
Hm, I think you should use `DataFrame.crossJoin` explicitly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748916198
@kozakana, I think you can specify empty column names in:
```scala
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String]): DataFrame
```
e.g.)
```scala
join(right, Nil)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on a change in pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on a change in pull request #30803:
URL: https://github.com/apache/spark/pull/30803#discussion_r545040023
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala
##########
@@ -102,7 +102,7 @@ case class NaturalJoin(tpe: JoinType) extends JoinType {
}
case class UsingJoin(tpe: JoinType, usingColumns: Seq[String]) extends JoinType {
- require(Seq(Inner, LeftOuter, LeftSemi, RightOuter, FullOuter, LeftAnti).contains(tpe),
+ require(Seq(Inner, LeftOuter, LeftSemi, RightOuter, FullOuter, LeftAnti, Cross).contains(tpe),
Review comment:
Does that mean it's better to modify the documentation?
[The join document](https://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.join) says:
```
how – str, default inner. Must be one of: inner, cross, outer, full, fullouter, full_outer, left, leftouter, left_outer, right, rightouter, right_outer, semi, leftsemi, left_semi, anti, leftanti and left_anti.
```
It says that cross is configurable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747873734
**[Test build #132990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132990/testReport)** for PR 30803 at commit [`87048bc`](https://github.com/apache/spark/commit/87048bccc23e02e64f5086e682146b16434f9b79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747873734
**[Test build #132990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132990/testReport)** for PR 30803 at commit [`87048bc`](https://github.com/apache/spark/commit/87048bccc23e02e64f5086e682146b16434f9b79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30803: [SPARK-33897][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30803:
URL: https://github.com/apache/spark/pull/30803
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748916198
@kozakana, I think you can specify empty column names in:
```scala
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
```
e.g.)
```scala
join(right, Nil, "cross")
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750028161
Thanks! I try it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748003804
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132990/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-746361252
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747893129
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37590/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747866018
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748849125
@kozakana do you mean the current behavior already makes sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748002716
**[Test build #132990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132990/testReport)** for PR 30803 at commit [`87048bc`](https://github.com/apache/spark/commit/87048bccc23e02e64f5086e682146b16434f9b79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747912395
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37590/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kozakana commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
kozakana commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750742951
ok! I understand the number and I can post the JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-747867672
cc @cloud-fan and @hvanhovell do you remember if we intentionally disabled cross join in `join` APIs? It's documented that `cross` is supported in `join` API. We should either fix the documentation or the codes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-33897][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-751330312
Merged to master and branch-3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-748916198
@kozakana, I think you can specify empty column names in:
```scala
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
```
e.g.)
```scala
join(right, Nil)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30803: [SPARK-30803][SQL]Can't set option 'cross' in join method.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30803:
URL: https://github.com/apache/spark/pull/30803#issuecomment-750726850
@kozakana, can you file a JIRA as guided in http://spark.apache.org/contributing.html? SPARK-30803 seems an unrelated JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org