You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/04/03 18:46:25 UTC

[jira] Created: (HBASE-559) MR example job to count table rows

MR example job to count table rows
----------------------------------

                 Key: HBASE-559
                 URL: https://issues.apache.org/jira/browse/HBASE-559
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack


The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-559:
------------------------

    Attachment: 559.patch

First cut at a RowCounter class.   Includes a 'Driver' class so we can plug in other MR jobs.  Need to hook this up as the Main-Class for the hbase jar so hbase jar can be a hadoop job jar.  Patch goes against branch for the moment (Lars wants a sample and he's running branch).

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-559:
------------------------

    Fix Version/s: 0.1.2
         Priority: Blocker  (was: Major)

Add to 1.1.2 (Its example/documentation).  Made it a blocker just so it'll be noticed.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-559) MR example job to count table rows

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589230#action_12589230 ] 

Jim Kellerman commented on HBASE-559:
-------------------------------------

If this is an example, shouldn't it live under src/examples?

in Driver.main, should (RowCounter.NAME, RowCounter.class,  "Count rows in HBase table") be hard-wired?

in RowCounter.main, should the master address be hard wired?
{code}
    c.set("hbase.master", "durruti.local:60000");
{code}

Otherwise, looks fine.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589234#action_12589234 ] 

stack commented on HBASE-559:
-----------------------------

I'll move RowCounter under examples when we have more than just one (Takes a bit of work setting up an example jar, etc.).  Also, this patch makes it so hbase.jar can be used as a job jar on a MR cluster.  Chose which job you want to run by passing switch to Driver class.  This facility doesn't belong in examples (but I need something to demo it with.... ).

In Driver.main, yeah, each new MR job is added here to the Driver class.

In RowCounter.main, I'll change it so its args[0] instead (This 'main' is not used generally).

Thanks for the review.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585402#action_12585402 ] 

stack commented on HBASE-559:
-----------------------------

To run you'd do something like the following (Presuming hbase including these new classes is in the hadoop CLASSPATH).

{code}
./bin/hadoop org.apache.hadoop.hbase.mapred.Driver rowcounter <outputdir> <tablename> <column1> [<column2>....]
{code}

The attached patch has not been properly tested.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-559) MR example job to count table rows

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589240#action_12589240 ] 

Jim Kellerman commented on HBASE-559:
-------------------------------------

+1

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-559.
-------------------------

    Resolution: Fixed

Committed to branch and trunk.  Added to the examples in the mapred javadoc and removed the hardcoding of my machine name suggested by Jim review.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reassigned HBASE-559:
---------------------------

    Assignee: stack

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-559) MR example job to count table rows

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-559:
------------------------

    Attachment: 559-0.1-v2.patch

Attached is a tested patch.  Also makes it so hbase.jar is now a hadoop job jar.   There's a Driver under mapred.  Add MR jobs to its list to make it so can do:

{code}
./bin/hadoop -jar hbase.jar
{code}

... and you'll get a list of the hbase MR jobs.

Here is how our dumb rowcounter looks:

{code}
durruti$ ./bin/hbase org.apache.hadoop.hbase.mapred.Driver rowcounter /tmp/output x x:
08/04/04 10:59:58 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
08/04/04 10:59:58 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/04/04 10:59:58 INFO mapred.JobClient: Running job: job_local_1
08/04/04 10:59:58 INFO mapred.MapTask: numReduceTasks: 1
08/04/04 10:59:58 INFO hbase.HTable: Creating scanner over x starting at key 
08/04/04 10:59:58 INFO mapred.LocalJobRunner: 
08/04/04 10:59:58 INFO mapred.TaskRunner: Task 'job_local_1_map_0000' done.
08/04/04 10:59:58 INFO mapred.LocalJobRunner: reduce > reduce
08/04/04 10:59:58 INFO mapred.TaskRunner: Task 'reduce_qv8ybc' done.
08/04/04 10:59:58 INFO mapred.TaskRunner: Saved output of task 'reduce_qv8ybc' to file:/tmp/output
08/04/04 10:59:59 INFO mapred.JobClient: Job complete: job_local_1
08/04/04 10:59:59 INFO mapred.JobClient: Counters: 10
08/04/04 10:59:59 INFO mapred.JobClient:   RowCounter
08/04/04 10:59:59 INFO mapred.JobClient:     Rows=1
08/04/04 10:59:59 INFO mapred.JobClient:   Map-Reduce Framework
08/04/04 10:59:59 INFO mapred.JobClient:     Map input records=1
08/04/04 10:59:59 INFO mapred.JobClient:     Map output records=1
08/04/04 10:59:59 INFO mapred.JobClient:     Map input bytes=0
08/04/04 10:59:59 INFO mapred.JobClient:     Map output bytes=7
08/04/04 10:59:59 INFO mapred.JobClient:     Combine input records=0
08/04/04 10:59:59 INFO mapred.JobClient:     Combine output records=0
08/04/04 10:59:59 INFO mapred.JobClient:     Reduce input groups=1
08/04/04 10:59:59 INFO mapred.JobClient:     Reduce input records=1
08/04/04 10:59:59 INFO mapred.JobClient:     Reduce output records=1
{code}

Here is commit comment:

{code}
M build.xml
    (Jar target): Add copying of any properties files under src/java
    Also added Main-Class to manifest.
M  src/java/org/apache/hadoop/hbase/mapred/RowCounter_Counters.properties
    Added resource so MR can print out counters for RowCounter MR job
M  src/java/org/apache/hadoop/hbase/mapred/RowCounter.java
    Example, simple MR job that counts non-empty rows.
M  src/java/org/apache/hadoop/hbase/mapred/Driver.java
    Driver class. General entry point for hbase MR jobs.
{code}

Will apply to branch after 0.1.1 goes out.  Will apply to TRUNK when MR API settles.

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-559) MR example job to count table rows

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HBASE-559:
--------------------------------

    Fix Version/s: 0.2.0

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>            Priority: Blocker
>             Fix For: 0.1.2, 0.2.0
>
>         Attachments: 559-0.1-v2.patch, 559.patch
>
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-559) MR example job to count table rows

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585193#action_12585193 ] 

Jim Kellerman commented on HBASE-559:
-------------------------------------

+1 great idea!

> MR example job to count table rows
> ----------------------------------
>
>                 Key: HBASE-559
>                 URL: https://issues.apache.org/jira/browse/HBASE-559
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> The Lars' import is a little messy; he's not sure how many records were imported.  Running a select takes a couple of hours.  He happens to have an idle MR cluster standing by.  An example MR job that just did a count of records would be generally useful.  Could even output row keys so you'd have a list of what made it in.   Later, if this tool becomes popular with derivatives and similiars, we can bundle a jar of MR jobs to run against your tables that can answer common queries and that are amenable to subclassing/modification.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.