You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mshtelma <gi...@git.apache.org> on 2018/03/27 13:53:57 UTC
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
GitHub user mshtelma opened a pull request:
https://github.com/apache/spark/pull/20913
[SPARK-23799] FilterEstimation.evaluateInSet produces devision by zero in a case of empty table with analyzed statistics
During evaluation of IN conditions, if the source table is empty, division by zero can occur. In order to fix this, check was added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mshtelma/spark filter_estimation_devision_by_zero
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20913.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20913
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88850/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88877/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88877/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88882/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #89099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89099/testReport)** for PR 20913 at commit [`5883c3c`](https://github.com/apache/spark/commit/5883c3c1fa64ed43d0e838aefcb9229f24c4a0e8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #89099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89099/testReport)** for PR 20913 at commit [`5883c3c`](https://github.com/apache/spark/commit/5883c3c1fa64ed43d0e838aefcb9229f24c4a0e8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by wzhfy <gi...@git.apache.org>.
Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/20913
@mshtelma Would you please update the branch? seems there's something wrong with the commits.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20913#discussion_r179322407
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala ---
@@ -427,7 +427,11 @@ case class FilterEstimation(plan: Filter) extends Logging {
// return the filter selectivity. Without advanced statistics such as histograms,
// we have to assume uniform distribution.
- Some(math.min(newNdv.toDouble / ndv.toDouble, 1.0))
+ if (ndv.toDouble != 0) {
--- End diff --
Can you add a test for the empty table case?
I think we need to fix the other places if they have the same issue. cc: @wzhfy
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20913
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20913
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88643/testReport)** for PR 20913 at commit [`5c8fe0e`](https://github.com/apache/spark/commit/5c8fe0e35bf70d6f85a12c0b8042102fd227bdde).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88934/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88934/testReport)** for PR 20913 at commit [`984faf5`](https://github.com/apache/spark/commit/984faf50ad055a770d190c8116fa427fec1cb983).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20913
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma commented on the issue:
https://github.com/apache/spark/pull/20913
@maropu done. The new one is here: https://github.com/apache/spark/pull/21052
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88643/testReport)** for PR 20913 at commit [`5c8fe0e`](https://github.com/apache/spark/commit/5c8fe0e35bf70d6f85a12c0b8042102fd227bdde).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20913
@mshtelma You shouldn't merge the changes and you need to rebase your pr. So, I think it's easy to close this pr and open a new pr as @jiangxb1987 suggested above.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88934/testReport)** for PR 20913 at commit [`984faf5`](https://github.com/apache/spark/commit/984faf50ad055a770d190c8116fa427fec1cb983).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88882/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88877/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88643/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma commented on a diff in the pull request:
https://github.com/apache/spark/pull/20913#discussion_r179063928
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala ---
@@ -427,7 +427,11 @@ case class FilterEstimation(plan: Filter) extends Logging {
// return the filter selectivity. Without advanced statistics such as histograms,
// we have to assume uniform distribution.
- Some(math.min(newNdv.toDouble / ndv.toDouble, 1.0))
+ if (ndv.toDouble != 0) {
--- End diff --
I have experienced this problem for the sub condition with IN clause, smth like "FLD in ("value")".
To my mind, this happens, if the table is empty. In this case ndv will be 0.
I think, it will make sense, to check it everywhere it is used in this way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88850/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20913
@mshtelma can you add a unit test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20913
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20913
@mshtelma You can close this PR and open a new one instead.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89099/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma commented on the issue:
https://github.com/apache/spark/pull/20913
I have added the unit case for this particular situation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma commented on the issue:
https://github.com/apache/spark/pull/20913
@jiangxb1987 @wzhfy Done. I have merged the latest changes from master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20913#discussion_r179037665
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala ---
@@ -427,7 +427,11 @@ case class FilterEstimation(plan: Filter) extends Logging {
// return the filter selectivity. Without advanced statistics such as histograms,
// we have to assume uniform distribution.
- Some(math.min(newNdv.toDouble / ndv.toDouble, 1.0))
+ if (ndv.toDouble != 0) {
--- End diff --
What's the concrete case when `ndv.toDouble == 0`?
Also, is this only an place where we need this check?
For example, we don't here:
https://github.com/apache/spark/blob/5cfd5fabcdbd77a806b98a6dd59b02772d2f6dee/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala#L166
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by apache-hivemall <gi...@git.apache.org>.
Github user apache-hivemall commented on the issue:
https://github.com/apache/spark/pull/20913
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20913
Good catch! +1 for adding tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma commented on a diff in the pull request:
https://github.com/apache/spark/pull/20913#discussion_r180343432
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala ---
@@ -427,7 +427,11 @@ case class FilterEstimation(plan: Filter) extends Logging {
// return the filter selectivity. Without advanced statistics such as histograms,
// we have to assume uniform distribution.
- Some(math.min(newNdv.toDouble / ndv.toDouble, 1.0))
+ if (ndv.toDouble != 0) {
--- End diff --
I have developed the test that illustrates the problem. It is just a small scala app right now.
Right now it breaks after the last select. Adding `REFRESH TABLE TBL` does not fix the problem in this particular case.
Should I add it to some particular test suite ?
object SparkCBOStatisticsTest {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder
.master("local[*]")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.cbo.enabled", "true")
.config("spark.sql.cbo.joinReorder.enabled", "true")
.config("spark.sql.sources.bucketing.enabled", "true")
.getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
val df = spark.range(10000L).select('id,
'id * 2 as "fld1",
'id * 12 as "fld2",
lit("aaa") + 'id as "fld3",
lit("bbb") + 'id * 'id as "fld4")
df.write
.mode(SaveMode.Overwrite)
.bucketBy(10, "id","fld1", "fld2")
.sortBy("id","fld1", "fld2")
.saveAsTable("TBL")
spark.sql("ANALYZE TABLE TBL COMPUTE STATISTICS ")
spark.sql("ANALYZE TABLE TBL COMPUTE STATISTICS FOR COLUMNS ID, FLD1, FLD2, FLD3")
val df2 = spark.sql("select t1.fld1*t2.fld2 as fld1, t1.fld2, t1.fld3 from tbl t1 join tbl t2 on t1.id=t2.id where t1.fld1>100 and t1.fld3 in (-123.23,321.23) ")
df2.explain(true)
df2.show(100)
df2.createTempView("tbl2")
spark.sql("DESC FORMATTED TBL").show(false)
spark.sql("select * from tbl2 where fld3 in ('qqq', 'qwe') and fld1<100 ").show(100, false)
}
}
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88771/testReport)** for PR 20913 at commit [`5c8fe0e`](https://github.com/apache/spark/commit/5c8fe0e35bf70d6f85a12c0b8042102fd227bdde).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by apache-hivemall <gi...@git.apache.org>.
Github user apache-hivemall commented on a diff in the pull request:
https://github.com/apache/spark/pull/20913#discussion_r179322297
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala ---
@@ -427,7 +427,11 @@ case class FilterEstimation(plan: Filter) extends Logging {
// return the filter selectivity. Without advanced statistics such as histograms,
// we have to assume uniform distribution.
- Some(math.min(newNdv.toDouble / ndv.toDouble, 1.0))
+ if (ndv.toDouble != 0) {
--- End diff --
Can you add a test for the empty table case?
I think we need to fix the other places if they have the same issue. cc: @wzhfy
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88882/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88850/testReport)** for PR 20913 at commit [`67597fd`](https://github.com/apache/spark/commit/67597fdcb703c7fa3fa189a456944693727d5754).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20913
**[Test build #88771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88771/testReport)** for PR 20913 at commit [`5c8fe0e`](https://github.com/apache/spark/commit/5c8fe0e35bf70d6f85a12c0b8042102fd227bdde).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88771/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20913: [SPARK-23799] FilterEstimation.evaluateInSet prod...
Posted by mshtelma <gi...@git.apache.org>.
Github user mshtelma closed the pull request at:
https://github.com/apache/spark/pull/20913
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20913: [SPARK-23799] FilterEstimation.evaluateInSet produces de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20913
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org