You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2019/01/04 02:51:52 UTC
[spark] branch master updated: [SPARK-26526][SQL][TEST] Fix invalid
test case about non-deterministic expression
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f65dc95 [SPARK-26526][SQL][TEST] Fix invalid test case about non-deterministic expression
f65dc95 is described below
commit f65dc9593ee4b84343fea04fdcace14096788be8
Author: Liu,Linhong <li...@baidu.com>
AuthorDate: Fri Jan 4 10:51:33 2019 +0800
[SPARK-26526][SQL][TEST] Fix invalid test case about non-deterministic expression
## What changes were proposed in this pull request?
Test case in SPARK-10316 is used to make sure non-deterministic `Filter` won't be pushed through `Project`
But in current code base this test case can't cover this purpose.
Change LogicalRDD to HadoopFsRelation can fix this issue.
## How was this patch tested?
Modified test pass.
Closes #23440 from LinhongLiu/fix-test.
Authored-by: Liu,Linhong <li...@baidu.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index b51c51e..3082e0b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -1398,11 +1398,14 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
}
test("SPARK-10316: respect non-deterministic expressions in PhysicalOperation") {
- val input = spark.read.json((1 to 10).map(i => s"""{"id": $i}""").toDS())
+ withTempDir { dir =>
+ (1 to 10).toDF("id").write.mode(SaveMode.Overwrite).json(dir.getCanonicalPath)
+ val input = spark.read.json(dir.getCanonicalPath)
- val df = input.select($"id", rand(0).as('r))
- df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === $"b.id").collect().foreach { row =>
- assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
+ val df = input.select($"id", rand(0).as('r))
+ df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === $"b.id").collect().foreach { row =>
+ assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org