You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Tim Robertson (JIRA)" <ji...@apache.org> on 2012/05/27 10:57:22 UTC
[jira] [Commented] (HBASE-6111) CLONE - Map tasks not local to RS
[ https://issues.apache.org/jira/browse/HBASE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284126#comment-13284126 ]
Tim Robertson commented on HBASE-6111:
--------------------------------------
I should note that this is also not consistent. Subsequent runs seems to get more data local map tasks, but the very last few always seem non local. Does anyone know if it is possible to force (almost force) JT to run data-local mappers?
> CLONE - Map tasks not local to RS
> ---------------------------------
>
> Key: HBASE-6111
> URL: https://issues.apache.org/jira/browse/HBASE-6111
> Project: HBase
> Issue Type: Bug
> Components: mapred, master, regionserver
> Affects Versions: 0.20.2, 0.92.0
> Environment: DN, TT and RS running on the same nodes, all using CDH3. Ganglia monitoring everything.
> Reporter: Tim Robertson
>
> I have started seeing this issue in our environment. HBASE-1672 was closed as non reproducible, so I cloned it here.
> I have a 367M record table, compressed with snappy, and running a vanilla MR SCAN with no filters spawns 441 Mappers. The cluster currently has 216 slots for mappers, and the first wave all report 100% data-local mappers. As the second wave of mappers come up they don't get run locally to the RS and data locality drops.
> This kills our environment, as it saturates the network at 120M which is very clear on ganglia.
> I am really happy to help diagnose this, but need some guidance on what to do. I don't know enough yet about how task assignment works in MR to determine why the machines are picking up random tasks for their second effort and not one for the local RS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira