You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/04/03 18:46:25 UTC
[jira] Created: (HBASE-559) MR example job to count table rows
MR example job to count table rows
----------------------------------
Key: HBASE-559
URL: https://issues.apache.org/jira/browse/HBASE-559
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-559:
------------------------
Attachment: 559.patch
First cut at a RowCounter class. Includes a 'Driver' class so we can plug in other MR jobs. Need to hook this up as the Main-Class for the hbase jar so hbase jar can be a hadoop job jar. Patch goes against branch for the moment (Lars wants a sample and he's running branch).
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-559:
------------------------
Fix Version/s: 0.1.2
Priority: Blocker (was: Major)
Add to 1.1.2 (Its example/documentation). Made it a blocker just so it'll be noticed.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-559) MR example job to count table rows
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589230#action_12589230 ]
Jim Kellerman commented on HBASE-559:
-------------------------------------
If this is an example, shouldn't it live under src/examples?
in Driver.main, should (RowCounter.NAME, RowCounter.class, "Count rows in HBase table") be hard-wired?
in RowCounter.main, should the master address be hard wired?
{code}
c.set("hbase.master", "durruti.local:60000");
{code}
Otherwise, looks fine.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589234#action_12589234 ]
stack commented on HBASE-559:
-----------------------------
I'll move RowCounter under examples when we have more than just one (Takes a bit of work setting up an example jar, etc.). Also, this patch makes it so hbase.jar can be used as a job jar on a MR cluster. Chose which job you want to run by passing switch to Driver class. This facility doesn't belong in examples (but I need something to demo it with.... ).
In Driver.main, yeah, each new MR job is added here to the Driver class.
In RowCounter.main, I'll change it so its args[0] instead (This 'main' is not used generally).
Thanks for the review.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585402#action_12585402 ]
stack commented on HBASE-559:
-----------------------------
To run you'd do something like the following (Presuming hbase including these new classes is in the hadoop CLASSPATH).
{code}
./bin/hadoop org.apache.hadoop.hbase.mapred.Driver rowcounter <outputdir> <tablename> <column1> [<column2>....]
{code}
The attached patch has not been properly tested.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-559) MR example job to count table rows
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589240#action_12589240 ]
Jim Kellerman commented on HBASE-559:
-------------------------------------
+1
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-559.
-------------------------
Resolution: Fixed
Committed to branch and trunk. Added to the examples in the mapred javadoc and removed the hardcoding of my machine name suggested by Jim review.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-559:
---------------------------
Assignee: stack
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-559) MR example job to count table rows
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-559:
------------------------
Attachment: 559-0.1-v2.patch
Attached is a tested patch. Also makes it so hbase.jar is now a hadoop job jar. There's a Driver under mapred. Add MR jobs to its list to make it so can do:
{code}
./bin/hadoop -jar hbase.jar
{code}
... and you'll get a list of the hbase MR jobs.
Here is how our dumb rowcounter looks:
{code}
durruti$ ./bin/hbase org.apache.hadoop.hbase.mapred.Driver rowcounter /tmp/output x x:
08/04/04 10:59:58 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
08/04/04 10:59:58 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/04/04 10:59:58 INFO mapred.JobClient: Running job: job_local_1
08/04/04 10:59:58 INFO mapred.MapTask: numReduceTasks: 1
08/04/04 10:59:58 INFO hbase.HTable: Creating scanner over x starting at key
08/04/04 10:59:58 INFO mapred.LocalJobRunner:
08/04/04 10:59:58 INFO mapred.TaskRunner: Task 'job_local_1_map_0000' done.
08/04/04 10:59:58 INFO mapred.LocalJobRunner: reduce > reduce
08/04/04 10:59:58 INFO mapred.TaskRunner: Task 'reduce_qv8ybc' done.
08/04/04 10:59:58 INFO mapred.TaskRunner: Saved output of task 'reduce_qv8ybc' to file:/tmp/output
08/04/04 10:59:59 INFO mapred.JobClient: Job complete: job_local_1
08/04/04 10:59:59 INFO mapred.JobClient: Counters: 10
08/04/04 10:59:59 INFO mapred.JobClient: RowCounter
08/04/04 10:59:59 INFO mapred.JobClient: Rows=1
08/04/04 10:59:59 INFO mapred.JobClient: Map-Reduce Framework
08/04/04 10:59:59 INFO mapred.JobClient: Map input records=1
08/04/04 10:59:59 INFO mapred.JobClient: Map output records=1
08/04/04 10:59:59 INFO mapred.JobClient: Map input bytes=0
08/04/04 10:59:59 INFO mapred.JobClient: Map output bytes=7
08/04/04 10:59:59 INFO mapred.JobClient: Combine input records=0
08/04/04 10:59:59 INFO mapred.JobClient: Combine output records=0
08/04/04 10:59:59 INFO mapred.JobClient: Reduce input groups=1
08/04/04 10:59:59 INFO mapred.JobClient: Reduce input records=1
08/04/04 10:59:59 INFO mapred.JobClient: Reduce output records=1
{code}
Here is commit comment:
{code}
M build.xml
(Jar target): Add copying of any properties files under src/java
Also added Main-Class to manifest.
M src/java/org/apache/hadoop/hbase/mapred/RowCounter_Counters.properties
Added resource so MR can print out counters for RowCounter MR job
M src/java/org/apache/hadoop/hbase/mapred/RowCounter.java
Example, simple MR job that counts non-empty rows.
M src/java/org/apache/hadoop/hbase/mapred/Driver.java
Driver class. General entry point for hbase MR jobs.
{code}
Will apply to branch after 0.1.1 goes out. Will apply to TRUNK when MR API settles.
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-559) MR example job to count table rows
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-559:
--------------------------------
Fix Version/s: 0.2.0
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.1.2, 0.2.0
>
> Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-559) MR example job to count table rows
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585193#action_12585193 ]
Jim Kellerman commented on HBASE-559:
-------------------------------------
+1 great idea!
> MR example job to count table rows
> ----------------------------------
>
> Key: HBASE-559
> URL: https://issues.apache.org/jira/browse/HBASE-559
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> The Lars' import is a little messy; he's not sure how many records were imported. Running a select takes a couple of hours. He happens to have an idle MR cluster standing by. An example MR job that just did a count of records would be generally useful. Could even output row keys so you'd have a list of what made it in. Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.