You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (Jira)" <ji...@apache.org> on 2020/06/23 05:17:00 UTC
[jira] [Updated] (SPARK-31584) NullPointerException when parsing
event log with InMemoryStore
[ https://issues.apache.org/jira/browse/SPARK-31584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-31584:
----------------------------
Fix Version/s: 3.1.0
> NullPointerException when parsing event log with InMemoryStore
> --------------------------------------------------------------
>
> Key: SPARK-31584
> URL: https://issues.apache.org/jira/browse/SPARK-31584
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.0.1
> Reporter: Baohe Zhang
> Assignee: Baohe Zhang
> Priority: Minor
> Fix For: 3.0.1, 3.1.0
>
> Attachments: errorstack.txt
>
>
> I compiled with the current branch-3.0 source and tested it in mac os. A java.lang.NullPointerException will be thrown when below conditions are met:
> # Using InMemoryStore as kvstore when parsing the event log file (e.g., when spark.history.store.path is unset).
> # At least one stage in this event log has task number greater than spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to delete extra task records.
> # The job has more than one stage, so parentToChildrenMap in InMemoryStore.java will have more than one key.
> The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the method deleteParentIndex().
> {code:java}
> private void deleteParentIndex(Object key) {
> if (hasNaturalParentIndex) {
> for (NaturalKeys v : parentToChildrenMap.values()) {
> if (v.remove(asKey(key))) {
> // `v` can be empty after removing the natural key and we can remove it from
> // `parentToChildrenMap`. However, `parentToChildrenMap` is a ConcurrentMap and such
> // checking and deleting can be slow.
> // This method is to delete one object with certain key, let's make it simple here.
> break;
> }
> }
> }
> }{code}
> In “if (v.remove(asKey(key)))”, if the key is not contained in v, "v.remove(asKey(key))" will return null, and java will throw a NullPointerException when executing "if (null)".
> An exception stack trace is attached.
> This issue can be fixed by updating if statement to
> {code:java}
> if (v.remove(asKey(key)) != null){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org