You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2017/09/21 17:36:43 UTC
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/19309
[SPARK-19558][sql] Add config key to register QueryExecutionListeners automatically.
This change adds a new SQL config key that is equivalent to SparkContext's
"spark.extraListeners", allowing users to register QueryExecutionListener
instances through the Spark configuration system instead of having to
explicitly do it in code.
The code used by SparkContext to implement the feature was refactored into
a helper method in the Utils class, and SQL's ExecutionListenerManager was
modified to use it to initialize listener declared in the configuration.
Unit tests were added to verify all the new functionality.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-19558
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19309.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19309
----
commit c52fd71899b136d79202eeb2c6502e2845b19714
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2017-09-20T22:58:23Z
[SPARK-19558][sql] Add config key to register QueryExecutionListeners automatically.
This change adds a new SQL config key that is equivalent to SparkContext's
"spark.extraListeners", allowing users to register QueryExecutionListener
instances through the Spark configuration system instead of having to
explicitly do it in code.
The code used by SparkContext to implement the feature was refactored into
a helper method in the Utils class, and SQL's ExecutionListenerManager was
modified to use it to initialize listener declared in the configuration.
Unit tests were added to verify all the new functionality.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82287/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19309#discussion_r141712571
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2687,6 +2688,59 @@ private[spark] object Utils extends Logging {
def stringToSeq(str: String): Seq[String] = {
str.split(",").map(_.trim()).filter(_.nonEmpty)
}
+
+ /**
+ * Create instances of extension classes.
+ *
+ * The classes in the given list must:
+ * - Be sub-classes of the given base class.
+ * - Provide either a no-arg constructor, or a 1-arg constructor that takes a SparkConf.
+ *
+ * The constructors are allowed to throw "UnsupportedOperationException" if the extension does not
+ * want to be registered; this allows the implementations to check the Spark configuration (or
+ * other state) and decide they do not need to be added. A log message is printed in that case.
+ * Other exceptions are bubbled up.
+ */
+ def loadExtensions[T](extClass: Class[T], classes: Seq[String], conf: SparkConf): Seq[T] = {
+ classes.flatMap { name =>
+ try {
+ val klass = classForName(name)
+ require(extClass.isAssignableFrom(klass),
+ s"$name is not a subclass of ${extClass.getName()}.")
+
+ val ext = Try(klass.getConstructor(classOf[SparkConf])) match {
+ case Success(ctor) =>
+ ctor.newInstance(conf)
+
+ case Failure(_) =>
+ klass.getConstructor().newInstance()
+ }
+
+ Some(ext.asInstanceOf[T])
+ } catch {
+ case _: NoSuchMethodException =>
+ throw new SparkException(
+ s"$name did not have a zero-argument constructor or a" +
+ " single-argument constructor that accepts SparkConf. Note: if the class is" +
+ " defined inside of another Scala class, then its constructors may accept an" +
+ " implicit parameter that references the enclosing class; in this case, you must" +
+ " define the class as a top-level class in order to prevent this extra" +
+ " parameter from breaking Spark's ability to find a valid constructor.")
+
+ case e: InvocationTargetException =>
+ e.getCause() match {
+ case _: UnsupportedOperationException =>
--- End diff --
I am a bit worried catching UnsupportedOperationException is a bit too general ... what if its thrown by something else deep inside the extension class? The user won't get to see what happened.
OTOH, I can't think of anything better to catch (doesn't seem worth a special exception type). Maybe log the entire exception at debug level?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19309
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82597/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19309
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19309
That's basically what the PR summary says.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82041/testReport)** for PR 19309 at commit [`c52fd71`](https://github.com/apache/spark/commit/c52fd71899b136d79202eeb2c6502e2845b19714).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19309
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82287/testReport)** for PR 19309 at commit [`bcc44e9`](https://github.com/apache/spark/commit/bcc44e9945f29f60a307bfa3a90225d4419622de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19309
Let me summarize the PR. Please correct me if anything is missing.
**Background**
Currently, our users can register a listener one by one during the executions by calling the SparkContext API:
> def addSparkListener(listener: SparkListenerInterface)
Users also can specify them through the Conf `spark.extraListeners`. SparkContext will automatically register the extra listeners during the initialization phase.
In Spark SQL, QueryExecutionListener enable users to register QueryExecutionListener by calling the API
> def register(listener: QueryExecutionListener)
**Proposal**
This PR is proposing to add a similar static SQLConf `spark.sql.queryExecutionListeners`. When a Spark session is built/cloned, the query execution listeners specified in the new conf will be registered automatically.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82041/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19309
/cc @gatorsmile @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19309#discussion_r143562613
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/util/ExecutionListenerManagerSuite.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.util
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.spark._
+import org.apache.spark.sql.execution.QueryExecution
+import org.apache.spark.sql.internal.StaticSQLConf._
+
+class ExecutionListenerManagerSuite extends SparkFunSuite {
+
+ import CountingQueryExecutionListener._
+
+ test("register query execution listeners using configuration") {
+ val conf = new SparkConf(false)
+ .set(QUERY_EXECUTION_LISTENERS, Seq(classOf[CountingQueryExecutionListener].getName()))
--- End diff --
That's covered by `UtilsSuite` already.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19309#discussion_r141715636
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2687,6 +2688,59 @@ private[spark] object Utils extends Logging {
def stringToSeq(str: String): Seq[String] = {
str.split(",").map(_.trim()).filter(_.nonEmpty)
}
+
+ /**
+ * Create instances of extension classes.
+ *
+ * The classes in the given list must:
+ * - Be sub-classes of the given base class.
+ * - Provide either a no-arg constructor, or a 1-arg constructor that takes a SparkConf.
+ *
+ * The constructors are allowed to throw "UnsupportedOperationException" if the extension does not
+ * want to be registered; this allows the implementations to check the Spark configuration (or
+ * other state) and decide they do not need to be added. A log message is printed in that case.
+ * Other exceptions are bubbled up.
+ */
+ def loadExtensions[T](extClass: Class[T], classes: Seq[String], conf: SparkConf): Seq[T] = {
+ classes.flatMap { name =>
+ try {
+ val klass = classForName(name)
+ require(extClass.isAssignableFrom(klass),
+ s"$name is not a subclass of ${extClass.getName()}.")
+
+ val ext = Try(klass.getConstructor(classOf[SparkConf])) match {
+ case Success(ctor) =>
+ ctor.newInstance(conf)
+
+ case Failure(_) =>
+ klass.getConstructor().newInstance()
+ }
+
+ Some(ext.asInstanceOf[T])
+ } catch {
+ case _: NoSuchMethodException =>
+ throw new SparkException(
+ s"$name did not have a zero-argument constructor or a" +
+ " single-argument constructor that accepts SparkConf. Note: if the class is" +
+ " defined inside of another Scala class, then its constructors may accept an" +
+ " implicit parameter that references the enclosing class; in this case, you must" +
+ " define the class as a top-level class in order to prevent this extra" +
+ " parameter from breaking Spark's ability to find a valid constructor.")
+
+ case e: InvocationTargetException =>
+ e.getCause() match {
+ case _: UnsupportedOperationException =>
--- End diff --
I wanted to avoid adding a special new exception for this. I'll add the log.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82597/testReport)** for PR 19309 at commit [`bcc44e9`](https://github.com/apache/spark/commit/bcc44e9945f29f60a307bfa3a90225d4419622de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19309#discussion_r143346651
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/util/ExecutionListenerManagerSuite.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.util
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import org.apache.spark._
+import org.apache.spark.sql.execution.QueryExecution
+import org.apache.spark.sql.internal.StaticSQLConf._
+
+class ExecutionListenerManagerSuite extends SparkFunSuite {
+
+ import CountingQueryExecutionListener._
+
+ test("register query execution listeners using configuration") {
+ val conf = new SparkConf(false)
+ .set(QUERY_EXECUTION_LISTENERS, Seq(classOf[CountingQueryExecutionListener].getName()))
--- End diff --
Now, this test only has one single listener whose constructor has zero arg. Could you add one more with a `SparkConf` arg in the constructor?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19309
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82597/testReport)** for PR 19309 at commit [`bcc44e9`](https://github.com/apache/spark/commit/bcc44e9945f29f60a307bfa3a90225d4419622de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82287/testReport)** for PR 19309 at commit [`bcc44e9`](https://github.com/apache/spark/commit/bcc44e9945f29f60a307bfa3a90225d4419622de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19309
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19309
Will review it this weekend.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19309
**[Test build #82041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82041/testReport)** for PR 19309 at commit [`c52fd71`](https://github.com/apache/spark/commit/c52fd71899b136d79202eeb2c6502e2845b19714).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` s\"$name is not a subclass of $`
* ` .doc(\"List of class names implementing QueryExecutionListener that will be automatically \" +`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org