You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/11/29 07:33:00 UTC
[jira] [Commented] (SPARK-30073) HistoryPage render "count" cost
too much time
[ https://issues.apache.org/jira/browse/SPARK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984782#comment-16984782 ]
Jungtaek Lim commented on SPARK-30073:
--------------------------------------
Are you playing with master branch hence set the affected version as 3.0? Because the stack trace is not in line with recent master, and I'm not seeing explicit call of count().
If you encounter this with master branch, it would be ideal to include the commit sha as well. If you're investigating this issue yourself, that would be great.
> HistoryPage render "count" cost too much time
> ---------------------------------------------
>
> Key: SPARK-30073
> URL: https://issues.apache.org/jira/browse/SPARK-30073
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 3.0.0
> Reporter: deshanxiao
> Priority: Major
>
> {code:java}
> "qtp1010584177-537" #537 daemon prio=5 os_prio=0 tid=0x00007f2734185000 nid=0x2c744 runnable [0x00007f23775e6000]
> java.lang.Thread.State: RUNNABLE
> at org.fusesource.leveldbjni.internal.NativeIterator$IteratorJNI.Prev(Native Method)
> at org.fusesource.leveldbjni.internal.NativeIterator.prev(NativeIterator.java:162)
> at org.fusesource.leveldbjni.internal.JniDBIterator.peekPrev(JniDBIterator.java:128)
> at org.fusesource.leveldbjni.internal.JniDBIterator.prev(JniDBIterator.java:144)
> at org.apache.spark.util.kvstore.LevelDBIterator.loadNext(LevelDBIterator.java:218)
> at org.apache.spark.util.kvstore.LevelDBIterator.hasNext(LevelDBIterator.java:111)
> at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.TraversableOnce$class.count(TraversableOnce.scala:115)
> at scala.collection.AbstractIterator.count(Iterator.scala:1336)
> at org.apache.spark.deploy.history.HistoryPage.render(HistoryPage.scala:50)
> at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:82)
> at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:82)
> at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
> at org.spark_project.jetty.server.handler.ContextHandler.do
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org