[jira] [Resolved] (SPARK-4206) BlockManager warnings in local mode: "Block $blockId already exists on this machine; not re-adding it

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-4206.
---------------------------------
    Resolution: Incomplete

> BlockManager warnings in local mode: "Block $blockId already exists on this machine; not re-adding it
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4206
>                 URL: https://issues.apache.org/jira/browse/SPARK-4206
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager
>         Environment: local mode, branch-1.1 & master
>            Reporter: Imran Rashid
>            Priority: Minor
>              Labels: bulk-closed
>
> When running in local mode, you often get log warning messages like:
> WARN storage.BlockManager: Block input-0-1415022975000 already exists on this machine; not re-adding it
> (eg., try running the TwitterPopularTags example in local mode)
> I think these warning messages are pretty unsettling for a new user, and should be removed.  If they are truly innocuous, they should be changed to logInfo, or maybe even logDebug.  Or if they might actually indicate a problem, we should find the root cause and fix it.
> I *think* the problem is caused by a replication level > 1 when running in local mode.  In BlockManager.doPut, first the block is put locally:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692
> and then if the replication level > 1, a request is sent out to replicate the block:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827
> However, in local mode, there isn't anywhere else to replicate the block; the request comes back to the same node, which then issues the warning that the block has already been added.
> If that analysis is right, the easy fix would be to make sure replicationLevel = 1 in local mode.  But, its a little disturbing that a replication request could result in an attempt to replicate on the same node -- and that if something is wrong, we only issue a warning and keep going.
> If this really the culprit, then it might be worth taking a closer look at the logic of replication.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org