You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/12/20 07:35:19 UTC
[GitHub] [incubator-kyuubi] cfmcgrady opened a new pull request #1591: Watchdog support for Spark-3.2
cfmcgrady opened a new pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#discussion_r772189454
##########
File path: dev/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.sql.watchdog
+
+import org.apache.spark.sql.catalyst.analysis.AnalysisContext
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreeNodeTag
+import org.apache.spark.sql.execution.command.DataWritingCommand
+
+import org.apache.kyuubi.sql.KyuubiSQLConf
+
+object ForcedMaxOutputRowsConstraint {
+ val CHILD_AGGREGATE: TreeNodeTag[String] = TreeNodeTag[String]("__kyuubi_child_agg__")
+ val CHILD_AGGREGATE_FLAG: String = "__kyuubi_child_agg__"
+}
+
+/*
+ * Add ForcedMaxOutputRows rule for output rows limitation
+ * to avoid huge output rows of non_limit query unexpectedly
+ * mainly applied to cases as below:
+ *
+ * case 1:
+ * {{{
+ * SELECT [c1, c2, ...]
+ * }}}
+ *
+ * case 2:
+ * {{{
+ * WITH CTE AS (
+ * ...)
+ * SELECT [c1, c2, ...] FROM CTE ...
+ * }}}
+ *
+ * The Logical Rule add a GlobalLimit node before root project
+ * */
+trait ForcedMaxOutputRowsBase extends Rule[LogicalPlan] {
+
+ protected def isChildAggregate(a: Aggregate): Boolean = a
+ .aggregateExpressions.exists(p =>
+ p.getTagValue(ForcedMaxOutputRowsConstraint.CHILD_AGGREGATE)
+ .contains(ForcedMaxOutputRowsConstraint.CHILD_AGGREGATE_FLAG))
+
+ protected def isView: Boolean = {
+ val nestedViewDepth = AnalysisContext.get.nestedViewDepth
+ nestedViewDepth > 0
+ }
+
+ protected def canInsertLimitInner(p: LogicalPlan): Boolean = p match {
+ case Aggregate(_, Alias(_, "havingCondition") :: Nil, _) => false
+ case agg: Aggregate => !isChildAggregate(agg)
+ case _: RepartitionByExpression => true
+ case _: Distinct => true
+ case _: Filter => true
+ case _: Project => true
+ case Limit(_, _) => true
+ case _: Sort => true
+ case Union(children, _, _) =>
+ if (children.exists(_.isInstanceOf[DataWritingCommand])) {
+ false
+ } else {
+ true
+ }
+ case _ => false
+ }
+
+ protected def canInsertLimit(p: LogicalPlan, maxOutputRowsOpt: Option[Int]): Boolean = {
+ maxOutputRowsOpt match {
+ case Some(forcedMaxOutputRows) => canInsertLimitInner(p) &&
+ !p.maxRows.exists(_ <= forcedMaxOutputRows) &&
+ !isView
+ case None => false
+ }
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ val maxOutputRowsOpt = conf.getConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS)
+ plan match {
+ case p if p.resolved && canInsertLimit(p, maxOutputRowsOpt) =>
+ Limit(
+ maxOutputRowsOpt.get,
+ plan)
+ case _ => plan
+ }
+ }
+
+}
+
+trait MarkAggregateOrderBase extends Rule[LogicalPlan] {
Review comment:
For Spark 3.2, we don't need `MarkAggregateOrderBase` about aggregate since https://github.com/apache/spark/pull/32470. We only need `CTE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-997710397
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1591](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364fc26) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/df1d9f3bb2a336054cb3f3e712c707a817612fdc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df1d9f3) will **decrease** coverage by `1.05%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head 364fc26 differs from pull request most recent head 0ce83ba. Consider uploading reports for the commit 0ce83ba to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1591 +/- ##
============================================
- Coverage 59.02% 57.97% -1.06%
+ Complexity 196 140 -56
============================================
Files 256 256
Lines 12708 12708
Branches 1601 1600 -1
============================================
- Hits 7501 7367 -134
- Misses 4570 4719 +149
+ Partials 637 622 -15
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9Gb3JjZWRNYXhPdXRwdXRSb3dzQmFzZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../kyuubi/sql/watchdog/KyuubiWatchDogException.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9LeXV1YmlXYXRjaERvZ0V4Y2VwdGlvbi5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9NYXhQYXJ0aXRpb25TdHJhdGVneS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/spark/sql/PruneFileSourcePartitionHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL1BydW5lRmlsZVNvdXJjZVBhcnRpdGlvbkhlbHBlci5zY2FsYQ==) | `0.00% <ø> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [df1d9f3...0ce83ba](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] cfmcgrady commented on a change in pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
cfmcgrady commented on a change in pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#discussion_r772183751
##########
File path: dev/kyuubi-extension-spark-common/src/test/scala/org/apache/spark/sql/WatchDogSuiteBase.scala
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.plans.logical.GlobalLimit
+
+import org.apache.kyuubi.sql.KyuubiSQLConf
+import org.apache.kyuubi.sql.watchdog.MaxPartitionExceedException
+
+trait WatchDogSuiteBase extends KyuubiSparkSQLExtensionTest {
+ override protected def beforeAll(): Unit = {
+ super.beforeAll()
+ setupData()
+ }
+
+ case class LimitAndExpected(limit: Int, expected: Int)
+ val limitAndExpecteds = List(LimitAndExpected(1, 1), LimitAndExpected(11, 10))
+
+ private def checkMaxPartition: Unit = {
+ withSQLConf(KyuubiSQLConf.WATCHDOG_MAX_PARTITIONS.key -> "100") {
+ checkAnswer(sql("SELECT count(distinct(p)) FROM test"), Row(10) :: Nil)
+ }
+ withSQLConf(KyuubiSQLConf.WATCHDOG_MAX_PARTITIONS.key -> "5") {
+ sql("SELECT * FROM test where p=1").queryExecution.sparkPlan
+
+ sql(s"SELECT * FROM test WHERE p in (${Range(0, 5).toList.mkString(",")})")
+ .queryExecution.sparkPlan
+
+ intercept[MaxPartitionExceedException](
+ sql("SELECT * FROM test where p != 1").queryExecution.sparkPlan)
+
+ intercept[MaxPartitionExceedException](
+ sql("SELECT * FROM test").queryExecution.sparkPlan)
+
+ intercept[MaxPartitionExceedException](sql(
+ s"SELECT * FROM test WHERE p in (${Range(0, 6).toList.mkString(",")})")
+ .queryExecution.sparkPlan)
+ }
+ }
+
+ test("watchdog with scan maxPartitions -- hive") {
+ Seq("textfile", "parquet").foreach { format =>
+ withTable("test", "temp") {
+ sql(
+ s"""
+ |CREATE TABLE test(i int)
+ |PARTITIONED BY (p int)
+ |STORED AS $format""".stripMargin)
+ spark.range(0, 10, 1).selectExpr("id as col")
+ .createOrReplaceTempView("temp")
+
+ for (part <- Range(0, 10)) {
+ sql(
+ s"""
+ |INSERT OVERWRITE TABLE test PARTITION (p='$part')
+ |select col from temp""".stripMargin)
+ }
+ checkMaxPartition
+ }
+ }
+ }
+
+ test("watchdog with scan maxPartitions -- data source") {
+ withTempDir { dir =>
+ withTempView("test") {
+ spark.range(10).selectExpr("id", "id as p")
+ .write
+ .partitionBy("p")
+ .mode("overwrite")
+ .save(dir.getCanonicalPath)
+ spark.read.load(dir.getCanonicalPath).createOrReplaceTempView("test")
+ checkMaxPartition
+ }
+ }
+ }
+
+ test("test watchdog: simple SELECT STATEMENT") {
+
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ List("", "ORDER BY c1", "ORDER BY c2").foreach { sort =>
+ List("", " DISTINCT").foreach { distinct =>
+ assert(sql(
+ s"""
+ |SELECT $distinct *
+ |FROM t1
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ List("", "ORDER BY c1", "ORDER BY c2").foreach { sort =>
+ List("", "DISTINCT").foreach { distinct =>
+ assert(sql(
+ s"""
+ |SELECT $distinct *
+ |FROM t1
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.analyzed.maxRows.contains(expected))
+ }
+ }
+ }
+ }
+ }
+
+ test("test watchdog: SELECT ... WITH AGGREGATE STATEMENT ") {
+
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ assert(!sql("SELECT count(*) FROM t1")
+ .queryExecution.analyzed.isInstanceOf[GlobalLimit])
+
+ val sorts = List("", "ORDER BY cnt", "ORDER BY c1", "ORDER BY cnt, c1", "ORDER BY c1, cnt")
+ val havingConditions = List("", "HAVING cnt > 1")
+
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |SELECT c1, COUNT(*) as cnt
+ |FROM t1
+ |GROUP BY c1
+ |$having
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |SELECT c1, COUNT(*) as cnt
+ |FROM t1
+ |GROUP BY c1
+ |$having
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.analyzed.maxRows.contains(expected))
+ }
+ }
+ }
+ }
+ }
+
+ test("test watchdog: SELECT with CTE forceMaxOutputRows") {
+ // simple CTE
+ val q1 =
+ """
+ |WITH t2 AS (
+ | SELECT * FROM t1
+ |)
+ |""".stripMargin
+
+ // nested CTE
+ val q2 =
+ """
+ |WITH
+ | t AS (SELECT * FROM t1),
+ | t2 AS (
+ | WITH t3 AS (SELECT * FROM t1)
+ | SELECT * FROM t3
+ | )
+ |""".stripMargin
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ val sorts = List("", "ORDER BY c1", "ORDER BY c2")
+
+ sorts.foreach { sort =>
+ Seq(q1, q2).foreach { withQuery =>
+ assert(sql(
+ s"""
+ |$withQuery
+ |SELECT * FROM t2
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ sorts.foreach { sort =>
+ Seq(q1, q2).foreach { withQuery =>
+ assert(sql(
+ s"""
+ |$withQuery
+ |SELECT * FROM t2
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.optimizedPlan.maxRows.contains(expected))
+ }
+ }
+ }
+ }
+ }
+
+ test("test watchdog: SELECT AGGREGATE WITH CTE forceMaxOutputRows") {
+
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ assert(!sql(
+ """
+ |WITH custom_cte AS (
+ |SELECT * FROM t1
+ |)
+ |
+ |SELECT COUNT(*)
+ |FROM custom_cte
+ |""".stripMargin).queryExecution
+ .analyzed.isInstanceOf[GlobalLimit])
+
+ val sorts = List("", "ORDER BY cnt", "ORDER BY c1", "ORDER BY cnt, c1", "ORDER BY c1, cnt")
+ val havingConditions = List("", "HAVING cnt > 1")
+
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |WITH custom_cte AS (
+ |SELECT * FROM t1
+ |)
+ |
+ |SELECT c1, COUNT(*) as cnt
+ |FROM custom_cte
+ |GROUP BY c1
+ |$having
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |WITH custom_cte AS (
+ |SELECT * FROM t1
+ |)
+ |
+ |SELECT c1, COUNT(*) as cnt
+ |FROM custom_cte
+ |GROUP BY c1
+ |$having
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.optimizedPlan.maxRows.contains(expected))
Review comment:
The same with https://github.com/apache/incubator-kyuubi/pull/1591/files#r772181897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-997710397
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1591](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364fc26) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/df1d9f3bb2a336054cb3f3e712c707a817612fdc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df1d9f3) will **decrease** coverage by `1.05%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head 364fc26 differs from pull request most recent head 5399a3f. Consider uploading reports for the commit 5399a3f to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1591 +/- ##
============================================
- Coverage 59.02% 57.97% -1.06%
+ Complexity 196 140 -56
============================================
Files 256 256
Lines 12708 12708
Branches 1601 1600 -1
============================================
- Hits 7501 7367 -134
- Misses 4570 4719 +149
+ Partials 637 622 -15
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9Gb3JjZWRNYXhPdXRwdXRSb3dzQmFzZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../kyuubi/sql/watchdog/KyuubiWatchDogException.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9LeXV1YmlXYXRjaERvZ0V4Y2VwdGlvbi5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9NYXhQYXJ0aXRpb25TdHJhdGVneS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/spark/sql/PruneFileSourcePartitionHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL1BydW5lRmlsZVNvdXJjZVBhcnRpdGlvbkhlbHBlci5zY2FsYQ==) | `0.00% <ø> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [df1d9f3...5399a3f](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-998458116
thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-997710397
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1591](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5399a3f) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/df1d9f3bb2a336054cb3f3e712c707a817612fdc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df1d9f3) will **decrease** coverage by `0.95%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1591 +/- ##
============================================
- Coverage 59.02% 58.06% -0.96%
+ Complexity 196 140 -56
============================================
Files 256 256
Lines 12708 12683 -25
Branches 1601 1596 -5
============================================
- Hits 7501 7365 -136
- Misses 4570 4695 +125
+ Partials 637 623 -14
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9Gb3JjZWRNYXhPdXRwdXRSb3dzQmFzZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../kyuubi/sql/watchdog/KyuubiWatchDogException.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9LeXV1YmlXYXRjaERvZ0V4Y2VwdGlvbi5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9NYXhQYXJ0aXRpb25TdHJhdGVneS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/spark/sql/PruneFileSourcePartitionHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL1BydW5lRmlsZVNvdXJjZVBhcnRpdGlvbkhlbHBlci5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...he/kyuubi/engine/spark/repl/KyuubiSparkILoop.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9yZXBsL0t5dXViaVNwYXJrSUxvb3Auc2NhbGE=) | `90.16% <0.00%> (-3.28%)` | :arrow_down: |
| [.../kyuubi/sql/watchdog/ForcedMaxOutputRowsRule.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9Gb3JjZWRNYXhPdXRwdXRSb3dzUnVsZS5zY2FsYQ==) | | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [df1d9f3...5399a3f](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-997710397
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1591](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (364fc26) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/df1d9f3bb2a336054cb3f3e712c707a817612fdc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df1d9f3) will **decrease** coverage by `1.05%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1591 +/- ##
============================================
- Coverage 59.02% 57.97% -1.06%
+ Complexity 196 140 -56
============================================
Files 256 256
Lines 12708 12708
Branches 1601 1600 -1
============================================
- Hits 7501 7367 -134
- Misses 4570 4719 +149
+ Partials 637 622 -15
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9Gb3JjZWRNYXhPdXRwdXRSb3dzQmFzZS5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [.../kyuubi/sql/watchdog/KyuubiWatchDogException.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9LeXV1YmlXYXRjaERvZ0V4Y2VwdGlvbi5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC93YXRjaGRvZy9NYXhQYXJ0aXRpb25TdHJhdGVneS5zY2FsYQ==) | `0.00% <ø> (ø)` | |
| [...che/spark/sql/PruneFileSourcePartitionHelper.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL1BydW5lRmlsZVNvdXJjZVBhcnRpdGlvbkhlbHBlci5zY2FsYQ==) | `0.00% <ø> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [df1d9f3...364fc26](https://codecov.io/gh/apache/incubator-kyuubi/pull/1591?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] cfmcgrady commented on a change in pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
cfmcgrady commented on a change in pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#discussion_r772181897
##########
File path: dev/kyuubi-extension-spark-common/src/test/scala/org/apache/spark/sql/WatchDogSuiteBase.scala
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.catalyst.plans.logical.GlobalLimit
+
+import org.apache.kyuubi.sql.KyuubiSQLConf
+import org.apache.kyuubi.sql.watchdog.MaxPartitionExceedException
+
+trait WatchDogSuiteBase extends KyuubiSparkSQLExtensionTest {
+ override protected def beforeAll(): Unit = {
+ super.beforeAll()
+ setupData()
+ }
+
+ case class LimitAndExpected(limit: Int, expected: Int)
+ val limitAndExpecteds = List(LimitAndExpected(1, 1), LimitAndExpected(11, 10))
+
+ private def checkMaxPartition: Unit = {
+ withSQLConf(KyuubiSQLConf.WATCHDOG_MAX_PARTITIONS.key -> "100") {
+ checkAnswer(sql("SELECT count(distinct(p)) FROM test"), Row(10) :: Nil)
+ }
+ withSQLConf(KyuubiSQLConf.WATCHDOG_MAX_PARTITIONS.key -> "5") {
+ sql("SELECT * FROM test where p=1").queryExecution.sparkPlan
+
+ sql(s"SELECT * FROM test WHERE p in (${Range(0, 5).toList.mkString(",")})")
+ .queryExecution.sparkPlan
+
+ intercept[MaxPartitionExceedException](
+ sql("SELECT * FROM test where p != 1").queryExecution.sparkPlan)
+
+ intercept[MaxPartitionExceedException](
+ sql("SELECT * FROM test").queryExecution.sparkPlan)
+
+ intercept[MaxPartitionExceedException](sql(
+ s"SELECT * FROM test WHERE p in (${Range(0, 6).toList.mkString(",")})")
+ .queryExecution.sparkPlan)
+ }
+ }
+
+ test("watchdog with scan maxPartitions -- hive") {
+ Seq("textfile", "parquet").foreach { format =>
+ withTable("test", "temp") {
+ sql(
+ s"""
+ |CREATE TABLE test(i int)
+ |PARTITIONED BY (p int)
+ |STORED AS $format""".stripMargin)
+ spark.range(0, 10, 1).selectExpr("id as col")
+ .createOrReplaceTempView("temp")
+
+ for (part <- Range(0, 10)) {
+ sql(
+ s"""
+ |INSERT OVERWRITE TABLE test PARTITION (p='$part')
+ |select col from temp""".stripMargin)
+ }
+ checkMaxPartition
+ }
+ }
+ }
+
+ test("watchdog with scan maxPartitions -- data source") {
+ withTempDir { dir =>
+ withTempView("test") {
+ spark.range(10).selectExpr("id", "id as p")
+ .write
+ .partitionBy("p")
+ .mode("overwrite")
+ .save(dir.getCanonicalPath)
+ spark.read.load(dir.getCanonicalPath).createOrReplaceTempView("test")
+ checkMaxPartition
+ }
+ }
+ }
+
+ test("test watchdog: simple SELECT STATEMENT") {
+
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ List("", "ORDER BY c1", "ORDER BY c2").foreach { sort =>
+ List("", " DISTINCT").foreach { distinct =>
+ assert(sql(
+ s"""
+ |SELECT $distinct *
+ |FROM t1
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ List("", "ORDER BY c1", "ORDER BY c2").foreach { sort =>
+ List("", "DISTINCT").foreach { distinct =>
+ assert(sql(
+ s"""
+ |SELECT $distinct *
+ |FROM t1
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.analyzed.maxRows.contains(expected))
+ }
+ }
+ }
+ }
+ }
+
+ test("test watchdog: SELECT ... WITH AGGREGATE STATEMENT ") {
+
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ assert(!sql("SELECT count(*) FROM t1")
+ .queryExecution.analyzed.isInstanceOf[GlobalLimit])
+
+ val sorts = List("", "ORDER BY cnt", "ORDER BY c1", "ORDER BY cnt, c1", "ORDER BY c1, cnt")
+ val havingConditions = List("", "HAVING cnt > 1")
+
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |SELECT c1, COUNT(*) as cnt
+ |FROM t1
+ |GROUP BY c1
+ |$having
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ havingConditions.foreach { having =>
+ sorts.foreach { sort =>
+ assert(sql(
+ s"""
+ |SELECT c1, COUNT(*) as cnt
+ |FROM t1
+ |GROUP BY c1
+ |$having
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.analyzed.maxRows.contains(expected))
+ }
+ }
+ }
+ }
+ }
+
+ test("test watchdog: SELECT with CTE forceMaxOutputRows") {
+ // simple CTE
+ val q1 =
+ """
+ |WITH t2 AS (
+ | SELECT * FROM t1
+ |)
+ |""".stripMargin
+
+ // nested CTE
+ val q2 =
+ """
+ |WITH
+ | t AS (SELECT * FROM t1),
+ | t2 AS (
+ | WITH t3 AS (SELECT * FROM t1)
+ | SELECT * FROM t3
+ | )
+ |""".stripMargin
+ withSQLConf(KyuubiSQLConf.WATCHDOG_FORCED_MAXOUTPUTROWS.key -> "10") {
+
+ val sorts = List("", "ORDER BY c1", "ORDER BY c2")
+
+ sorts.foreach { sort =>
+ Seq(q1, q2).foreach { withQuery =>
+ assert(sql(
+ s"""
+ |$withQuery
+ |SELECT * FROM t2
+ |$sort
+ |""".stripMargin).queryExecution.analyzed.isInstanceOf[GlobalLimit])
+ }
+ }
+
+ limitAndExpecteds.foreach { case LimitAndExpected(limit, expected) =>
+ sorts.foreach { sort =>
+ Seq(q1, q2).foreach { withQuery =>
+ assert(sql(
+ s"""
+ |$withQuery
+ |SELECT * FROM t2
+ |$sort
+ |LIMIT $limit
+ |""".stripMargin).queryExecution.optimizedPlan.maxRows.contains(expected))
Review comment:
As `WithCTE.maxRows` is `None`, we should check `optimizedPlan` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] cfmcgrady commented on pull request #1591: Watchdog support for Spark-3.2
Posted by GitBox <gi...@apache.org>.
cfmcgrady commented on pull request #1591:
URL: https://github.com/apache/incubator-kyuubi/pull/1591#issuecomment-997713982
cc @ulysses-you and watchdog original author @zhouyifan279 @i7xh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org