You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (Commented) (JIRA)" <ji...@apache.org> on 2012/01/14 00:06:39 UTC
[jira] [Commented] (HBASE-5166) MultiThreaded Table Mapper
analogous to MultiThreaded Mapper in hadoop
[ https://issues.apache.org/jira/browse/HBASE-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185963#comment-13185963 ]
stack commented on HBASE-5166:
------------------------------
bq. Moreover, I want to know whether It would be a good/bad idea to use HBase for these kind of usecases ?.
Looks grand to me (as does the network/io-bound justification in your usecase). Would be a nice contrib. I'd like it so I can use it putting up load on hbase; currently have to run a ridiculous amount of concurrent mappers putting up a load using a tool like PerformanceEvaluation which runs a single client doing serial load per map task.
A few comments on the patch.
No need of these lines:
{code}
+ * Copyright 2007 The Apache Software Foundation
{code}
In our code base, we use two spaces for tabs (no hard tabs you have in your file).
Fix the name of this config:
{code}
+ getInt("mapred.map.multithreadedrunner.threads", 10);
{code}
Ditto for the setter.
You don't want to use an executor and something like guava's utility creating the executor running the threads? (See hbase code base for examples)
> MultiThreaded Table Mapper analogous to MultiThreaded Mapper in hadoop
> ----------------------------------------------------------------------
>
> Key: HBASE-5166
> URL: https://issues.apache.org/jira/browse/HBASE-5166
> Project: HBase
> Issue Type: Improvement
> Reporter: Jai Kumar Singh
> Priority: Minor
> Labels: multithreaded, tablemapper
> Attachments: 0001-Added-MultithreadedTableMapper-HBASE-5166.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> There is no MultiThreadedTableMapper in hbase currently just like we have a MultiThreadedMapper in Hadoop for IO Bound Jobs.
> UseCase, webcrawler: take input (urls) from a hbase table and put the content (urls, content) back into hbase.
> Running these kind of hbase mapreduce job with normal table mapper is quite slow as we are not utilizing CPU fully (N/W IO Bound).
> Moreover, I want to know whether It would be a good/bad idea to use HBase for these kind of usecases ?.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira