PerformanceEvaluation test doesn't flush within timer

[Bob Jacoby <bo...@cabsmail.com>]

I'm using HBase 0.20.3. I'm trying to calculate some performance numbers 
for our Hbase setup. I'm using the test suite included with HBase as a 
starting point. However, looking at the test code, the timer showing how 
long the test took is stopped prior to performing an explicit flush of 
the table. This would appear to mean that the contents of the write 
buffer at the end of the test will never be written to HBase, which 
would significantly affect the results of the test if the buffer is only 
filled a small number of times (or never filled).

Shouldn't the flush occur after the loop in the test() method, but 
before the elapsedTime is calculated? Am I missing something in how 
these tests are run?

org.apache.hadoop.hbase.PerformanceEvaluation$Test test() method is the 
following:

    long test() throws IOException {
      long elapsedTime;
      testSetup();
      long startTime = System.currentTimeMillis();
      try {
        int lastRow = this.startRow + this.perClientRunRows;
        // Report on completion of 1/10th of total.
        for (int i = this.startRow; i < lastRow; i++) {
          testRow(i);
          if (status != null && i > 0 && (i % getReportingPeriod()) == 0) {
            status.setStatus(generateStatus(this.startRow, i, lastRow));
          }
        }
        elapsedTime = System.currentTimeMillis() - startTime;
      } finally {
        testTakedown();
      }
      return elapsedTime;
    }

    void testTakedown()  throws IOException {
      this.table.flushCommits();
    }

The sequentialWrite testRow method is as following:
    void testRow(final int i) throws IOException {
      Put put = new Put(format(i));
      put.add(FAMILY_NAME, QUALIFIER_NAME, generateValue(this.rand));
      table.put(put);
    }

Thanks,
Bob

Re: PerformanceEvaluation test doesn't flush within timer

[Dan Washusen <da...@reactive.org>]

You are correct.  It will flush the commits after 12MB of data has been
written to the write buffer.  However, it doesn't take the last batch into
account...

On 9 March 2010 07:03, Bob Jacoby <bo...@cabsmail.com> wrote:

> I'm using HBase 0.20.3. I'm trying to calculate some performance numbers
> for our Hbase setup. I'm using the test suite included with HBase as a
> starting point. However, looking at the test code, the timer showing how
> long the test took is stopped prior to performing an explicit flush of the
> table. This would appear to mean that the contents of the write buffer at
> the end of the test will never be written to HBase, which would
> significantly affect the results of the test if the buffer is only filled a
> small number of times (or never filled).
>
> Shouldn't the flush occur after the loop in the test() method, but before
> the elapsedTime is calculated? Am I missing something in how these tests are
> run?
>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test test() method is the
> following:
>
>   long test() throws IOException {
>     long elapsedTime;
>     testSetup();
>     long startTime = System.currentTimeMillis();
>     try {
>       int lastRow = this.startRow + this.perClientRunRows;
>       // Report on completion of 1/10th of total.
>       for (int i = this.startRow; i < lastRow; i++) {
>         testRow(i);
>         if (status != null && i > 0 && (i % getReportingPeriod()) == 0) {
>           status.setStatus(generateStatus(this.startRow, i, lastRow));
>         }
>       }
>       elapsedTime = System.currentTimeMillis() - startTime;
>     } finally {
>       testTakedown();
>     }
>     return elapsedTime;
>   }
>
>   void testTakedown()  throws IOException {
>     this.table.flushCommits();
>   }
>
> The sequentialWrite testRow method is as following:
>   void testRow(final int i) throws IOException {
>     Put put = new Put(format(i));
>     put.add(FAMILY_NAME, QUALIFIER_NAME, generateValue(this.rand));
>     table.put(put);
>   }
>
> Thanks,
> Bob
>