You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Imran Rashid (JIRA)" <ji...@apache.org> on 2014/11/03 15:24:35 UTC
[jira] [Updated] (SPARK-4206) BlockManager warnings in local mode:
"Block $blockId already exists on this machine; not re-adding it
[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Imran Rashid updated SPARK-4206:
--------------------------------
Description:
When running in local mode, you often get log warning messages like:
WARN storage.BlockManager: Block input-0-1415022975000 already exists on this machine; not re-adding it
(eg., try running the TwitterPopularTags example in local mode)
I think these warning messages are pretty unsettling for a new user, and should be removed. If they are truly innocuous, they should be changed to logInfo, or maybe even logDebug. Or if they might actually indicate a problem, we should find the root cause and fix it.
I *think* the problem is caused by a replication level > 1 when running in local mode. In BlockManager.doPut, first the block is put locally:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692
and then if the replication level > 1, a request is sent out to replicate the block:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827
However, in local mode, there isn't anywhere else to replicate the block; the request comes back to the same node, which then issues the warning that the block has already been added.
If that analysis is right, the easy fix would be to make sure replicationLevel = 1 in local mode. But, its a little disturbing that a replication request could result in an attempt to replicate on the same node -- and that if something is wrong, we only issue a warning and keep going.
If this really the culprit, then it might be worth taking a closer look at the logic of replication.
Environment: local mode, branch-1.1 & master
> BlockManager warnings in local mode: "Block $blockId already exists on this machine; not re-adding it
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-4206
> URL: https://issues.apache.org/jira/browse/SPARK-4206
> Project: Spark
> Issue Type: Bug
> Environment: local mode, branch-1.1 & master
> Reporter: Imran Rashid
> Priority: Minor
>
> When running in local mode, you often get log warning messages like:
> WARN storage.BlockManager: Block input-0-1415022975000 already exists on this machine; not re-adding it
> (eg., try running the TwitterPopularTags example in local mode)
> I think these warning messages are pretty unsettling for a new user, and should be removed. If they are truly innocuous, they should be changed to logInfo, or maybe even logDebug. Or if they might actually indicate a problem, we should find the root cause and fix it.
> I *think* the problem is caused by a replication level > 1 when running in local mode. In BlockManager.doPut, first the block is put locally:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L692
> and then if the replication level > 1, a request is sent out to replicate the block:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L827
> However, in local mode, there isn't anywhere else to replicate the block; the request comes back to the same node, which then issues the warning that the block has already been added.
> If that analysis is right, the easy fix would be to make sure replicationLevel = 1 in local mode. But, its a little disturbing that a replication request could result in an attempt to replicate on the same node -- and that if something is wrong, we only issue a warning and keep going.
> If this really the culprit, then it might be worth taking a closer look at the logic of replication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org