You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Vu Tan (Jira)" <ji...@apache.org> on 2022/05/02 05:19:00 UTC
[jira] [Updated] (SPARK-39082) Fix FsHistoryProvider race condition between update and clean app data
[ https://issues.apache.org/jira/browse/SPARK-39082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vu Tan updated SPARK-39082:
---------------------------
Fix Version/s: (was: 3.3.0)
(was: 3.2.1)
> Fix FsHistoryProvider race condition between update and clean app data
> ----------------------------------------------------------------------
>
> Key: SPARK-39082
> URL: https://issues.apache.org/jira/browse/SPARK-39082
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.1.2, 3.2.1, 3.3.0
> Reporter: Vu Tan
> Assignee: XiDuo You
> Priority: Major
>
> After SPARK-29043, FsHistoryProvider will list the log info without waitting all `mergeApplicationListing` task finished.
> However the `LevelDBIterator` of list log info is not thread safe if some other threads delete the related log info at same time.
> There is the error msg:
> {code:java}
> 21/12/15 14:12:02 ERROR FsHistoryProvider: Exception in checking for event log updates
> java.util.NoSuchElementException: 1^@__main__^@+hdfs://xxx/application_xxx.inprogress
> at org.apache.spark.util.kvstore.LevelDB.get(LevelDB.java:132)
> at org.apache.spark.util.kvstore.LevelDBIterator.next(LevelDBIterator.java:137)
> at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
> at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
> at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:184)
> at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:47)
> at scala.collection.TraversableLike.to(TraversableLike.scala:678)
> at scala.collection.TraversableLike.to$(TraversableLike.scala:675)
> at scala.collection.AbstractTraversable.to(Traversable.scala:108)
> at scala.collection.TraversableOnce.toList(TraversableOnce.scala:299)
> at scala.collection.TraversableOnce.toList$(TraversableOnce.scala:299)
> at scala.collection.AbstractTraversable.toList(Traversable.scala:108)
> at org.apache.spark.deploy.history.FsHistoryProvider.checkForLogs(FsHistoryProvider.scala:588)
> at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$startPolling$3(FsHistoryProvider.scala:299)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org