You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by an...@apache.org on 2015/06/10 22:21:56 UTC
spark git commit: [SPARK-7756] CORE RDDOperationScope fix for IBM Java
Repository: spark
Updated Branches:
refs/heads/master 30ebf1a23 -> 19e30b48f
[SPARK-7756] CORE RDDOperationScope fix for IBM Java
IBM Java has an extra method when we do getStackTrace(): this is "getStackTraceImpl", a native method. This causes two tests to fail within "DStreamScopeSuite" when running with IBM Java. Instead of "map" or "filter" being the method names found, "getStackTrace" is returned. This commit addresses such an issue by using dropWhile. Given that our current method is withScope, we look for the next method that isn't ours: we don't care about methods that come before us in the stack trace: e.g. getStackTrace (regardless of how many levels this might go).
IBM:
java.lang.Thread.getStackTraceImpl(Native Method)
java.lang.Thread.getStackTrace(Thread.java:1117)
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:104)
Oracle:
PRINTING STACKTRACE!!!
java.lang.Thread.getStackTrace(Thread.java:1552)
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:106)
I've tested this with Oracle and IBM Java, no side effects for other tests introduced.
Author: Adam Roberts <ar...@uk.ibm.com>
Author: a-roberts <ar...@uk.ibm.com>
Closes #6740 from a-roberts/RDDScopeStackCrawlFix and squashes the following commits:
13ce390 [Adam Roberts] Ensure consistency with String equality checking
a4fc0e0 [a-roberts] Update RDDOperationScope.scala
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/19e30b48
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/19e30b48
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/19e30b48
Branch: refs/heads/master
Commit: 19e30b48f3c6d0b72871d3e15b9564c1b2822700
Parents: 30ebf1a
Author: Adam Roberts <ar...@uk.ibm.com>
Authored: Wed Jun 10 13:21:01 2015 -0700
Committer: Andrew Or <an...@databricks.com>
Committed: Wed Jun 10 13:21:51 2015 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/rdd/RDDOperationScope.scala | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/19e30b48/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala b/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
index 6b09dfa..4466728 100644
--- a/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala
@@ -95,10 +95,9 @@ private[spark] object RDDOperationScope extends Logging {
private[spark] def withScope[T](
sc: SparkContext,
allowNesting: Boolean = false)(body: => T): T = {
- val stackTrace = Thread.currentThread.getStackTrace().tail // ignore "Thread#getStackTrace"
- val ourMethodName = stackTrace(1).getMethodName // i.e. withScope
- // Climb upwards to find the first method that's called something different
- val callerMethodName = stackTrace
+ val ourMethodName = "withScope"
+ val callerMethodName = Thread.currentThread.getStackTrace()
+ .dropWhile(_.getMethodName != ourMethodName)
.find(_.getMethodName != ourMethodName)
.map(_.getMethodName)
.getOrElse {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org