You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lins05 <gi...@git.apache.org> on 2016/10/06 08:02:34 UTC
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
GitHub user lins05 opened a pull request:
https://github.com/apache/spark/pull/15377
[SPARK-17802] Improved caller context logging.
## What changes were proposed in this pull request?
[SPARK-16757](https://issues.apache.org/jira/browse/SPARK-16757) sets the hadoop `CallerContext` when calling hadoop/hdfs apis to make spark applications more diagnosable in hadoop/hdfs logs. However, the `org.apache.hadoop.ipc.CallerContext` class is only added since [hadoop 2.8](https://issues.apache.org/jira/browse/HDFS-9184). So each time `utils.CallerContext.setCurrentContext()` is called (e.g [when a task is created](https://github.com/apache/spark/blob/b678e46/core/src/main/scala/org/apache/spark/scheduler/Task.scala#L95-L96)), a "java.lang.ClassNotFoundException: org.apache.hadoop.ipc.CallerContext"
error is logged, which pollutes the spark logs when there are lots of tasks.
This patch improves this behaviour by only logging the `ClassNotFoundException` once.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lins05/spark spark-17802-improve-callercontext-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15377
----
commit b8621ab361966c6a63015ae08b0f63985db25176
Author: Shuai Lin <li...@gmail.com>
Date: 2016-10-06T03:18:20Z
[SPARK-17802] Improved caller context logging.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66719/consoleFull)** for PR 15377 at commit [`df28bdd`](https://github.com/apache/spark/commit/df28bdddce5e4789a02cf7ef5dedab8b7c408630).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82136454
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ var callerContextSupported : Option[Boolean] = _
--- End diff --
Why not just Boolean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66506/consoleFull)** for PR 15377 at commit [`f1962e4`](https://github.com/apache/spark/commit/f1962e41981c80bb366a8a2060edadee59f95022).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66449/consoleFull)** for PR 15377 at commit [`bc2fb25`](https://github.com/apache/spark/commit/bc2fb25a4350b4ee7b395c6bbdd35701a3f2c3c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/15377
Another thing, do you verify it locally? Since there's no unit test to cover it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67201/consoleFull)** for PR 15377 at commit [`b3b794c`](https://github.com/apache/spark/commit/b3b794c4e41dc390a52b65edfabe4b176b7b6147).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66506/consoleFull)** for PR 15377 at commit [`f1962e4`](https://github.com/apache/spark/commit/f1962e41981c80bb366a8a2060edadee59f95022).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
Utils#classForName will make maven build fail, please do not use it before figure out : https://issues.apache.org/jira/browse/SPARK-17714. The failure is here: https://issues.apache.org/jira/browse/SPARK-17710 and .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
@lins05 Maybe something like this:
```
test("Set Spark CallerContext") {
val context = "test"
try {
new CallerContext(context).setCurrentContext()
val callerContext = Utils.classForName("org.apache.hadoop.ipc.CallerContext")
assert(CallerContext.callerContextSupported)
assert(s"SPARK_$context" ===
callerContext.getMethod("getCurrent").invoke(null).toString)
} catch {
case e: ClassNotFoundException =>
assert(!CallerContext.callerContextSupported)
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
@srowen done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r84241076
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,18 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext {
+ val callerContextSupported: Boolean = {
+ SparkHadoopUtil
--- End diff --
Let's find a different way to wrap this; it's non-standard and hard to read the condition here. Maybe break out `conf` as a local `val` to make it simpler.
I see why you're using Try, though I wonder if we are suppressing potentially important errors this way. For example, ClassNotFoundException obviously means 'not supported' and that's normal. A fatal error should probably just propagate. But a NonFatal error could be logged with a warning? because that would be unexpected. It would still mean the caller context isn't supported rather than cause a failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67239/consoleFull)** for PR 15377 at commit [`9cffcf3`](https://github.com/apache/spark/commit/9cffcf3755e9f1dc31477f042ee246dabf2c6740).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82181784
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ private var callerContextSupported: Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e: ClassNotFoundException =>
+ logInfo(
+ s"Fail to set Spark caller context: requires Hadoop 2.8 or later: ${e.getMessage}")
+ callerContextSupported = false
+ false
+ case NonFatal(e) =>
+ logWarning("Fail to set Spark caller context", e)
--- End diff --
I guess I mean: set `callerContextSupported = false` here too right? or else you're still going to fail over and over and log it if something besides ClassNotFoundException happens. Which is probably a pretty fatal error actually, but hey.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66507/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83217975
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
--- End diff --
Here, it won't matter if it's called multiple times. It just means some extra work is done but the result is the same. The flag is at least local to the class here. I don't know of a better way to record this state because it is global, and properly so, as the things it depends on can't vary within one JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66556/consoleFull)** for PR 15377 at commit [`dc6951d`](https://github.com/apache/spark/commit/dc6951d925305419412c9f769aa423006028dc2b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66891/consoleFull)** for PR 15377 at commit [`e83e101`](https://github.com/apache/spark/commit/e83e101a8272a10f90f0119e40f37a29e9ec505b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66885/consoleFull)** for PR 15377 at commit [`884de64`](https://github.com/apache/spark/commit/884de64d5910bbd7517f834807c78d6590e688b5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66440/consoleFull)** for PR 15377 at commit [`b8621ab`](https://github.com/apache/spark/commit/b8621ab361966c6a63015ae08b0f63985db25176).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66449/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82492566
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ private var callerContextSupported: Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e: ClassNotFoundException =>
+ logInfo(
+ s"Fail to set Spark caller context: requires Hadoop 2.8 or later: ${e.getMessage}")
+ callerContextSupported = false
+ false
+ case NonFatal(e) =>
+ logWarning("Fail to set Spark caller context", e)
--- End diff --
Thanks for the tip about `hadoop.caller.context.enabled` and the suggestion, i'll do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67124/consoleFull)** for PR 15377 at commit [`b745394`](https://github.com/apache/spark/commit/b745394d97a40bdc9891ba029dc1b14f013df52e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82942829
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
--- End diff --
Please use `SparkHadoopUtils#conf`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67124/consoleFull)** for PR 15377 at commit [`b745394`](https://github.com/apache/spark/commit/b745394d97a40bdc9891ba029dc1b14f013df52e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
Thanks @jerryshao @srowen . I've updated the code like what you suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82136515
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ var callerContextSupported : Option[Boolean] = _
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ callerContextSupported match {
+ case Some(false) =>
+ false
+ case _ =>
+ var succeed = false
--- End diff --
This was also superfluous, just end the method with the right return value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67201/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83178194
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,10 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext {
+ var callerContextSupported: Boolean = true
--- End diff --
It's used below, and you have commented on it :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66886/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15377
I think you can avoid it for now, and if that other issue is addressed, it can go back and use the `Utils` method again everywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82141925
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,35 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ var callerContextSupported : Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = Builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e : ClassNotFoundException =>
+ logInfo("Fail to set Spark caller context: requires hadoop 2.8 or later.", e)
+ callerContextSupported = false
+ false
--- End diff --
Just put `false` at the end of the method like before. Do you really want to log the exception at info level? it looks like a problem when it's not. Nits: Builder -> builder, hadoop -> Hadoop. Don't put a space before the colons, and the new callerContextSupported flag should be private.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82139283
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ var callerContextSupported : Option[Boolean] = _
--- End diff --
Makes sense. I have simplied the logic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66556/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67394/consoleFull)** for PR 15377 at commit [`c93bed3`](https://github.com/apache/spark/commit/c93bed3838a1ce56c4340a9fe7a708ac8c76bcca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67201/consoleFull)** for PR 15377 at commit [`b3b794c`](https://github.com/apache/spark/commit/b3b794c4e41dc390a52b65edfabe4b176b7b6147).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
@weiqingy Emm, then we would also add the logic of checking "hadoop.caller.context.enabled" in the test code, which makes the test code simply duplicates the code path of `CallerContext.callerContextSupported`, and IMHO this doesn't make much sense...
On the other hand I do agree we should test the code path, but it seems not easy to do it.
@srowen @jerryshao What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/15377
@srowen I'm not against this change, personally because the usage of flag is wired to me and frankly saying I haven't seen such pattern in the Spark code.
Since we want to avoid re-executing the code if CallContext is not enabled or not existed. We could get this information immutably like:
```scala
private[util] object CallerContext {
val callerContextSupported: Boolean = if (callContext is not enabled) {
false
} else if (callContext is not existed) {
false
} else {
true
}
}
```
And in `setCurrentContext` we could get rid of this mutable things, use the flag to decide whether to set the caller context without reset flag. That will simplify the codes and make others easy to understand.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
The test case wanted to test \u201cif it\u2019s running on a hadoop version that supports caller context, our code does set it\u201d. But in this change , if somehow the flag was set incorrectly (being false instead of true), it would\u2019t be able to find out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66443/consoleFull)** for PR 15377 at commit [`3b5f9e4`](https://github.com/apache/spark/commit/3b5f9e46e01157e65fe388f0547e29bb012d31a6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67447/consoleFull)** for PR 15377 at commit [`5acefad`](https://github.com/apache/spark/commit/5acefaddb065b550ad48a2c680ff98dbf50b117e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83179598
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83179592
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
+ false
+ } else {
+ if (!conf.getBoolean("hadoop.caller.context.enabled", false)) {
+ logInfo("Hadoop caller context is not enabled")
+ CallerContext.callerContextSupported = false
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
> Another thing, do you verify it locally? Since there's no unit test to cover it.
@jerryshao Yeah, I did test it locally to ensure the error is only logged once.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83340292
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
--- End diff --
If you cannot fully preventing re-executing this code, and to the worst all threads will executing the same logics again, so is it necessary enough to add such flag? For me I think it is some kind of undeterministic that will confuse the user (some tasks printed the log while others not).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/15377
Yup, that's what I mean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/15377
LGTM just some minor things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66447/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67039/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15377
@weiqingy all the Maven/SBT builds pass now, are you sure this is still an issue? I know the PR builder is just using SBT, yes, but thought we had fixed this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
Thanks for proposing this PR. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82441129
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ private var callerContextSupported: Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e: ClassNotFoundException =>
+ logInfo(
+ s"Fail to set Spark caller context: requires Hadoop 2.8 or later: ${e.getMessage}")
+ callerContextSupported = false
+ false
+ case NonFatal(e) =>
+ logWarning("Fail to set Spark caller context", e)
--- End diff --
There's some different deployment situations here.
1. you are running on Hadoop 2.8, want caller context. A failure here is something to mention.
1. you are running on Hadoop <= 2.7, don't want context or care about it. Here another stack trace is going to be a distraction; if it's not a support call then it gets added to the list of "error messages you learn to ignore". (this is my current state, BTW)
1. you want caller context, but are running on an incompatible version of Hadoop. Again, here, logging the CNFE makes sense.
Question is: do you need anything if the caller context is disabled? As I don't see you do. And there's a Hadoop config option `hadoop.caller.context.enabled` (default false), which controls that.
What about looking for the config option, if it is set going through the introspection work, reporting problems with stack traces. And if unset: don't even bother with the introspection?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15377
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r84643899
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2508,6 +2508,27 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext extends Logging {
+ val callerContextSupported: Boolean = {
+ val conf = SparkHadoopUtil.get.conf
+ conf.getBoolean("hadoop.caller.context.enabled", false) && {
--- End diff --
Although this construct looks funny to me, a more conventional structure would just be a little longer. I'm OK with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #3372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3372/consoleFull)** for PR 15377 at commit [`5acefad`](https://github.com/apache/spark/commit/5acefaddb065b550ad48a2c680ff98dbf50b117e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66715/consoleFull)** for PR 15377 at commit [`7485ffa`](https://github.com/apache/spark/commit/7485ffaa3df508f35df4b878ed715eb1ece0f4db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
@srowen Could we get this merged since the tests are now green? Not sure why it failed previously, it just turned green without me doing anything.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67394/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66885/consoleFull)** for PR 15377 at commit [`884de64`](https://github.com/apache/spark/commit/884de64d5910bbd7517f834807c78d6590e688b5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66447/consoleFull)** for PR 15377 at commit [`112ddcb`](https://github.com/apache/spark/commit/112ddcb9eb34c12baf90ae0ad97fdea3f7a2e304).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66447/consoleFull)** for PR 15377 at commit [`112ddcb`](https://github.com/apache/spark/commit/112ddcb9eb34c12baf90ae0ad97fdea3f7a2e304).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67447/consoleFull)** for PR 15377 at commit [`5acefad`](https://github.com/apache/spark/commit/5acefaddb065b550ad48a2c680ff98dbf50b117e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66556/consoleFull)** for PR 15377 at commit [`dc6951d`](https://github.com/apache/spark/commit/dc6951d925305419412c9f769aa423006028dc2b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67039/consoleFull)** for PR 15377 at commit [`80c64d5`](https://github.com/apache/spark/commit/80c64d50c3795e2a2f249aba876150fb48f37d97).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66506/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
@weiqingy I agree that's a problem. But i don't see how to unit test the `callerContextSupported` method without repeating the same logic in the test code. Do you have any suggestion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82492524
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,10 @@ private[spark] object Utils extends Logging {
}
}
+private[spark] object CallerContext {
+ var callerContextSupported: Boolean = true
--- End diff --
Makes sense, done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82159342
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,34 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ private var callerContextSupported: Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e: ClassNotFoundException =>
+ logInfo("Fail to set Spark caller context: requires Hadoop 2.8 or later.", e)
--- End diff --
I still think you perhaps should not log the exception but just log the message (so that the caller sees what class was not found, like `logInfo(s"... or later: ${e.getMessage}")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
Not sure why you need to check hadoop.caller.context.enabled" since just need to make callerContextClassExists be access by test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67445/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66715/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66886/consoleFull)** for PR 15377 at commit [`c42019b`](https://github.com/apache/spark/commit/c42019b16d133095c18e7dfe47f55e16e464ea52).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82139291
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ var callerContextSupported : Option[Boolean] = _
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ callerContextSupported match {
+ case Some(false) =>
+ false
+ case _ =>
+ var succeed = false
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67445/consoleFull)** for PR 15377 at commit [`b57f009`](https://github.com/apache/spark/commit/b57f009b853e770a7c8fb47367677b9d259b1981).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82224216
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2474,36 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ private var callerContextSupported: Boolean = true
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!callerContextSupported) {
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
+ val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
+ val builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ val builderInst = builder.getConstructor(classOf[String]).newInstance(context)
+ val hdfsContext = builder.getMethod("build").invoke(builderInst)
+ callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
+ true
+ } catch {
+ case e: ClassNotFoundException =>
+ logInfo(
+ s"Fail to set Spark caller context: requires Hadoop 2.8 or later: ${e.getMessage}")
+ callerContextSupported = false
+ false
+ case NonFatal(e) =>
+ logWarning("Fail to set Spark caller context", e)
--- End diff --
Emm, I prefer not to suppress the non-ClassNotFound errors, because they are real errors, compared to the ClassNotFound error, which is rather a *conditional feature* based on the hadoop environment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67239/consoleFull)** for PR 15377 at commit [`9cffcf3`](https://github.com/apache/spark/commit/9cffcf3755e9f1dc31477f042ee246dabf2c6740).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82943152
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
+ false
+ } else {
+ if (!conf.getBoolean("hadoop.caller.context.enabled", false)) {
+ logInfo("Hadoop caller context is not enabled")
+ CallerContext.callerContextSupported = false
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
--- End diff --
Nit: indent is not correct, use 2 ws.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66449/consoleFull)** for PR 15377 at commit [`bc2fb25`](https://github.com/apache/spark/commit/bc2fb25a4350b4ee7b395c6bbdd35701a3f2c3c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66443/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15377
@jerryshao can I take your temperature about how against this change you are? maybe @lins05 can elaborate again on what this is preventing, and what happens in case of a race between two threads. That is, what do we expect in the average case before and after?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67394/consoleFull)** for PR 15377 at commit [`c93bed3`](https://github.com/apache/spark/commit/c93bed3838a1ce56c4340a9fe7a708ac8c76bcca).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by weiqingy <gi...@git.apache.org>.
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/15377
Before 28/Sep/16, using Utils#classForName could pass the PR builder, but would fail the Spark QA tests in master Maven builds. Unit now (from 28/Sep/16 - now), if there is no change in maven build of Spark QA test, I think using Utils#classForName here will still fail the Spark QA tests in master Maven builds.
There is the related discussion about this (the last part of conversations): https://github.com/apache/spark/pull/14659#issuecomment-248402339)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83179622
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
+ false
+ } else {
+ if (!conf.getBoolean("hadoop.caller.context.enabled", false)) {
+ logInfo("Hadoop caller context is not enabled")
+ CallerContext.callerContextSupported = false
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66507/consoleFull)** for PR 15377 at commit [`f6557d6`](https://github.com/apache/spark/commit/f6557d6e2fd79cf778966b0665fdd493a0de0b0e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66891/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
cc @weiqingy (who worked on [SPARK-16757](https://issues.apache.org/jira/browse/SPARK-16757).)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66440/consoleFull)** for PR 15377 at commit [`b8621ab`](https://github.com/apache/spark/commit/b8621ab361966c6a63015ae08b0f63985db25176).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67368/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83181601
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2479,20 +2483,35 @@ private[spark] class CallerContext(
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
+ false
+ } else {
+ if (!SparkHadoopUtil.get.conf.getBoolean("hadoop.caller.context.enabled", false)) {
+ logInfo("Hadoop caller context is not enabled")
+ CallerContext.callerContextSupported = false
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
--- End diff --
Remove this comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15377
Yeah, you're saying all the checks can be done upfront? I agree. It will need a tiny bit of code duplication but is more robust. I think we could also potentially get rid of the return value from `setCallerContext`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66719/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66719/consoleFull)** for PR 15377 at commit [`df28bdd`](https://github.com/apache/spark/commit/df28bdddce5e4789a02cf7ef5dedab8b7c408630).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66891/consoleFull)** for PR 15377 at commit [`e83e101`](https://github.com/apache/spark/commit/e83e101a8272a10f90f0119e40f37a29e9ec505b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66715/consoleFull)** for PR 15377 at commit [`7485ffa`](https://github.com/apache/spark/commit/7485ffaa3df508f35df4b878ed715eb1ece0f4db).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67039/consoleFull)** for PR 15377 at commit [`80c64d5`](https://github.com/apache/spark/commit/80c64d50c3795e2a2f249aba876150fb48f37d97).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67368/consoleFull)** for PR 15377 at commit [`af5a95b`](https://github.com/apache/spark/commit/af5a95ba86aeb686071e4f3e5144b43e0a7f4ba1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r84573004
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,30 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext extends Logging {
+ private val callerContextClassExists = {
+ try {
+ // scalastyle:off classforname
+ Class.forName("org.apache.hadoop.ipc.CallerContext")
+ Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ true
+ } catch {
+ case _: ClassNotFoundException =>
+ false
+ case NonFatal(e) =>
+ logWarning("Fail to load the CallerContext class", e)
+ false
+ }
+ }
+
+ private val conf = SparkHadoopUtil.get.conf
--- End diff --
This doesn't need to be a separate member; it can be a local val.
Still, why have more than one member here? Initializing `callerContextSupported` in one block of code is sufficient and simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67445/consoleFull)** for PR 15377 at commit [`b57f009`](https://github.com/apache/spark/commit/b57f009b853e770a7c8fb47367677b9d259b1981).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/15377
@weiqingy @srowen I see. So do you suggest to avoid using `Utils.classForName` to get this one merged, or rather wait for SPARK-17714?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66440/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67124/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r84212380
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,26 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext {
+ private def callerContextClassExists(): Boolean = {
+ try {
+ // scalastyle:off classforname
+ Class.forName("org.apache.hadoop.ipc.CallerContext")
+ Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ true
+ } catch {
+ case _ : ClassNotFoundException => false
+ }
+ }
--- End diff --
The above code can be simplified with `Try { xxxx }.isSuccess`, please check `scala.util.Try`. Then these two methods can be merged into one expression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66443/consoleFull)** for PR 15377 at commit [`3b5f9e4`](https://github.com/apache/spark/commit/3b5f9e46e01157e65fe388f0547e29bb012d31a6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82428699
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,10 @@ private[spark] object Utils extends Logging {
}
}
+private[spark] object CallerContext {
+ var callerContextSupported: Boolean = true
--- End diff --
OK, could it even be `private[util]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83206341
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2479,20 +2483,35 @@ private[spark] class CallerContext(
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
+ false
+ } else {
+ if (!SparkHadoopUtil.get.conf.getBoolean("hadoop.caller.context.enabled", false)) {
+ logInfo("Hadoop caller context is not enabled")
+ CallerContext.callerContextSupported = false
+ false
+ } else {
+ try {
+ // scalastyle:off classforname
--- End diff --
Good catch, done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66886/consoleFull)** for PR 15377 at commit [`c42019b`](https://github.com/apache/spark/commit/c42019b16d133095c18e7dfe47f55e16e464ea52).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r84097831
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,26 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext {
+ private def callerContextClassExists(): Boolean = {
+ try {
+ // scalastyle:off classforname
+ Class.forName("org.apache.hadoop.ipc.CallerContext")
+ Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
+ // scalastyle:on classforname
+ true
+ } catch {
+ case _ : ClassNotFoundException => false
+ }
+ }
+
+ private def callerContextEnabled(): Boolean = {
+ SparkHadoopUtil.get.conf.getBoolean("hadoop.caller.context.enabled", false)
+ }
+
+ lazy val callerContextSupported: Boolean = callerContextEnabled && callerContextClassExists
--- End diff --
I don't think this needs to be lazy (?) and doesn't need to be broken out into methods. It can just be one `val` initialized directly upfront in one expression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67447/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #3372 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3372/consoleFull)** for PR 15377 at commit [`5acefad`](https://github.com/apache/spark/commit/5acefaddb065b550ad48a2c680ff98dbf50b117e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r82942438
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2432,6 +2432,10 @@ private[spark] object Utils extends Logging {
}
}
+private[util] object CallerContext {
+ var callerContextSupported: Boolean = true
--- End diff --
What is the usage of this flag? I don't see any other place use it, all just setters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #66507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66507/consoleFull)** for PR 15377 at commit [`f6557d6`](https://github.com/apache/spark/commit/f6557d6e2fd79cf778966b0665fdd493a0de0b0e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15377
**[Test build #67368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67368/consoleFull)** for PR 15377 at commit [`af5a95b`](https://github.com/apache/spark/commit/af5a95ba86aeb686071e4f3e5144b43e0a7f4ba1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/15377
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66885/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15377: [SPARK-17802] Improved caller context logging.
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/15377#discussion_r83336101
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2474,25 +2478,42 @@ private[spark] class CallerContext(
val context = "SPARK_" + from + appIdStr + appAttemptIdStr +
jobIdStr + stageIdStr + stageAttemptIdStr + taskIdStr + taskAttemptNumberStr
+ lazy val conf = new Configuration
+
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
* [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
*/
def setCurrentContext(): Boolean = {
- var succeed = false
- try {
- // scalastyle:off classforname
- val callerContext = Class.forName("org.apache.hadoop.ipc.CallerContext")
- val Builder = Class.forName("org.apache.hadoop.ipc.CallerContext$Builder")
- // scalastyle:on classforname
- val builderInst = Builder.getConstructor(classOf[String]).newInstance(context)
- val hdfsContext = Builder.getMethod("build").invoke(builderInst)
- callerContext.getMethod("setCurrent", callerContext).invoke(null, hdfsContext)
- succeed = true
- } catch {
- case NonFatal(e) => logInfo("Fail to set Spark caller context", e)
+ if (!CallerContext.callerContextSupported) {
--- End diff --
Alright, so the main purpose to use this flag is to avoid re-executing the codes below. So AFAIK unless we can guarantee the sequentially calling of this method, otherwise `callerContextSupported` may introduce concurrent issue, especially in `Task` running code. Simultaneously started running tasks will possibly get the flag as `true` and executing below code, which makes this flag useless.
Also in AM/Client, it will only be called once, so this flag is not used at those JVMs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15377: [SPARK-17802] Improved caller context logging.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67239/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org