You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yan Xiaole (Jira)" <ji...@apache.org> on 2020/08/08 12:03:00 UTC
[jira] [Created] (SPARK-32574) Race condition in FsHistoryProvider
listing iteration
Yan Xiaole created SPARK-32574:
----------------------------------
Summary: Race condition in FsHistoryProvider listing iteration
Key: SPARK-32574
URL: https://issues.apache.org/jira/browse/SPARK-32574
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Yan Xiaole
It seems to be a race condition in FSHistoryProvider that iterate the listing with high concurrency:
{code:java}
val stale = listing.view(classOf[LogInfo])
.index("lastProcessed")
.last(newLastScanTime - 1)
.asScala
.toList{code}
`toList` will iterate items in the listing, if one of listing entries was deleted by some clean thread, checkForLogs will throw out a `java.util.NoSuchElementException` and abort current execution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org