You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2018/12/17 17:46:59 UTC
[spark] branch master updated: [SPARK-26255][YARN] Apply user
provided UI filters to SQL tab in yarn mode
This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6d45e6e [SPARK-26255][YARN] Apply user provided UI filters to SQL tab in yarn mode
6d45e6e is described below
commit 6d45e6ea1507943f6ee833af8ad7969294b0356a
Author: chakravarthi <tc...@gmail.com>
AuthorDate: Mon Dec 17 09:46:50 2018 -0800
[SPARK-26255][YARN] Apply user provided UI filters to SQL tab in yarn mode
## What changes were proposed in this pull request?
User specified filters are not applied to SQL tab in yarn mode, as it is overridden by the yarn AmIp filter.
So we need to append user provided filters (spark.ui.filters) with yarn filter.
## How was this patch tested?
【Test step】：
1) Launch spark sql with authentication filter as below:
2) spark-sql --master yarn --conf spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter --conf spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"
3) Go to Yarn application list UI link
4) Launch the application master for the Spark-SQL app ID and access all the tabs by appending tab name.
5) It will display an error for all tabs including SQL tab.(before able to access SQL tab,as Authentication filter is not applied for SQL tab)
6) Also can be verified with info logs,that Authentication filter applied to SQL tab.(before it is not applied).
I have attached the behaviour below in following order..
1) Command used
2) Before fix (logs and UI)
3) After fix (logs and UI)
**1) COMMAND USED**:
launching spark-sql with authentication filter.
**2) BEFORE FIX:**
**UI result:**
able to access SQL tab.
**logs**:
authentication filter not applied to SQL tab.
**3) AFTER FIX:**
**UI result**:
Not able to access SQL tab.
**in logs**:
Both yarn filter and Authentication filter applied to SQL tab.
Closes #23312 from chakravarthiT/SPARK-26255_ui.
Authored-by: chakravarthi <tc...@gmail.com>
Signed-off-by: Marcelo Vanzin <va...@cloudera.com>
---
.../org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
index 67c36aa..1289d4b 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
@@ -168,8 +168,10 @@ private[spark] abstract class YarnSchedulerBackend(
filterName != null && filterName.nonEmpty &&
filterParams != null && filterParams.nonEmpty
if (hasFilter) {
+ // SPARK-26255: Append user provided filters(spark.ui.filters) with yarn filter.
+ val allFilters = filterName + "," + conf.get("spark.ui.filters", "")
logInfo(s"Add WebUI Filter. $filterName, $filterParams, $proxyBase")
- conf.set("spark.ui.filters", filterName)
+ conf.set("spark.ui.filters", allFilters)
filterParams.foreach { case (k, v) => conf.set(s"spark.$filterName.param.$k", v) }
scheduler.sc.ui.foreach { ui => JettyUtils.addFilters(ui.getHandlers, conf) }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org