You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@accumulo.apache.org by Bill Havanki <bh...@clouderagovt.com> on 2014/06/26 23:52:10 UTC

Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/
-----------------------------------------------------------

Review request for accumulo and Bill Slacum.

Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947

Repository: accumulo

Description
-------

A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.

The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.

Diffs
-----

test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
test/system/stress/stress-env.sh.example 1360c67
test/system/stress/writer.sh 7d9b283

Diff: https://reviews.apache.org/r/23093/diff/

Testing
-------

Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.

Thanks,

Bill Havanki

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.


> On June 30, 2014, 4:05 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/Write.java, lines 87-88
> > <https://reviews.apache.org/r/23093/diff/1/?file=618989#file618989line87>
> >
> >     Might just be RB, but the indent here looks off.
> 
> Bill Havanki wrote:
>     The argument "opts.max_cells_per_mutation" should be at the same indent level as the first "new RandomWithinRange" above it and the "new RandomByteArrays" above that; they are all arguments to the RandomMutations constructor. The close paren after "max_cells_per_mutation" ends the arg list for that constructor, and the ");" on the next line closes the call to "new DataWriter".
>     
>     It looks like what I have in my editor in RB ... so given all that, is it OK?
> 
> Mike Drob wrote:
>     Ah, yes, I was misreading, and thought that this was an argument to the previous constructor.

Yes. Much tabs.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47012
-----------------------------------------------------------


On July 1, 2014, 12:23 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 12:23 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Mike Drob <md...@mdrob.com>.


> On June 30, 2014, 8:05 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/Write.java, lines 87-88
> > <https://reviews.apache.org/r/23093/diff/1/?file=618989#file618989line87>
> >
> >     Might just be RB, but the indent here looks off.
> 
> Bill Havanki wrote:
>     The argument "opts.max_cells_per_mutation" should be at the same indent level as the first "new RandomWithinRange" above it and the "new RandomByteArrays" above that; they are all arguments to the RandomMutations constructor. The close paren after "max_cells_per_mutation" ends the arg list for that constructor, and the ");" on the next line closes the call to "new DataWriter".
>     
>     It looks like what I have in my editor in RB ... so given all that, is it OK?

Ah, yes, I was misreading, and thought that this was an argument to the previous constructor.


- Mike


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47012
-----------------------------------------------------------


On July 1, 2014, 4:23 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 4:23 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.


> On June 30, 2014, 4:05 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/Write.java, lines 87-88
> > <https://reviews.apache.org/r/23093/diff/1/?file=618989#file618989line87>
> >
> >     Might just be RB, but the indent here looks off.

The argument "opts.max_cells_per_mutation" should be at the same indent level as the first "new RandomWithinRange" above it and the "new RandomByteArrays" above that; they are all arguments to the RandomMutations constructor. The close paren after "max_cells_per_mutation" ends the arg list for that constructor, and the ");" on the next line closes the call to "new DataWriter".

It looks like what I have in my editor in RB ... so given all that, is it OK?


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47012
-----------------------------------------------------------


On June 26, 2014, 5:52 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated June 26, 2014, 5:52 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Mike Drob <md...@mdrob.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47012
-----------------------------------------------------------



test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java
<https://reviews.apache.org/r/23093/#comment82612>

    This check could be part of the loop condition.



test/src/main/java/org/apache/accumulo/test/stress/random/Write.java
<https://reviews.apache.org/r/23093/#comment82611>

    Might just be RB, but the indent here looks off.



test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java
<https://reviews.apache.org/r/23093/#comment82609>

    Please document that a non-positive value implies unbounded, infinity, or MAX_INT (verbiage up to you).


- Mike Drob


On June 26, 2014, 9:52 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated June 26, 2014, 9:52 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Josh Elser <jo...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47128
-----------------------------------------------------------

Ship it!


Ship It!

- Josh Elser


On July 1, 2014, 7 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 7 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.

(Updated July 1, 2014, 3 p.m.)

Review request for accumulo and Bill Slacum.

Changes
-------

Incorporated Josh's feedback.

Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947

Repository: accumulo

Description
-------

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/23093/diff/

Testing
-------

Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.

Thanks,

Bill Havanki

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.


> On July 1, 2014, 1:43 p.m., Josh Elser wrote:
> > test/system/stress/stress-env.sh.example, line 55
> > <https://reviews.apache.org/r/23093/diff/3/?file=621403#file621403line55>
> >
> >     Would be nice to include what the default -1 value equates to in the comment (in addition to the help message) since you have a nice comment describing the argument already.

Good deal.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47105
-----------------------------------------------------------


On July 1, 2014, 3 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 3 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Josh Elser <jo...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47105
-----------------------------------------------------------



test/system/stress/stress-env.sh.example
<https://reviews.apache.org/r/23093/#comment82710>

    Would be nice to include what the default -1 value equates to in the comment (in addition to the help message) since you have a nice comment describing the argument already.


- Josh Elser


On July 1, 2014, 5:09 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 5:09 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.

(Updated July 1, 2014, 1:09 p.m.)

Review request for accumulo and Bill Slacum.

Changes
-------

More of Mike's feedback.

Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947

Repository: accumulo

Description
-------

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/23093/diff/

Testing
-------

Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.

Thanks,

Bill Havanki

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Mike Drob <md...@mdrob.com>.


> On July 1, 2014, 4:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 53
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line53>
> >
> >     I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.
> 
> Bill Havanki wrote:
>     I hadn't thought of that, actually ... it just turned out that way. At this stage, I would be fine with it failing horribly if a put() throws an exception. In that case, I can actually just say this once:
>     
>     cells_remaining_in_row -= cells;
>     
>     How's that?

Sure, as long as you do it after the loop.


- Mike


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------


On July 1, 2014, 4:23 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 4:23 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.


> On July 1, 2014, 12:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 50
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line50>
> >
> >     I don't know why, but I like this. I would have done (i < max && i < remaining), but this is much cleaner. :)

I did have something like (i < max && i < remaining) at first, but realized the whole thing could be precomputed since max and remaining are invariant.


> On July 1, 2014, 12:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 53
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line53>
> >
> >     I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.

I hadn't thought of that, actually ... it just turned out that way. At this stage, I would be fine with it failing horribly if a put() throws an exception. In that case, I can actually just say this once:

cells_remaining_in_row -= cells;

How's that?


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------


On July 1, 2014, 12:23 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 12:23 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Mike Drob <md...@mdrob.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------



test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java
<https://reviews.apache.org/r/23093/#comment82681>

    I don't know why, but I like this. I would have done (i < max && i < remaining), but this is much cleaner. :)



test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java
<https://reviews.apache.org/r/23093/#comment82679>

    I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.


- Mike Drob


On July 1, 2014, 4:23 p.m., Bill Havanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
> 
> (Updated July 1, 2014, 4:23 p.m.)
> 
> 
> Review request for accumulo and Bill Slacum.
> 
> 
> Bugs: ACCUMULO-2947
>     https://issues.apache.org/jira/browse/ACCUMULO-2947
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
> 
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
> 
> 
> Diffs
> -----
> 
>   test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2 
>   test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871 
>   test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528 
>   test/system/stress/stress-env.sh.example 1360c67 
>   test/system/stress/writer.sh 7d9b283 
> 
> Diff: https://reviews.apache.org/r/23093/diff/
> 
> 
> Testing
> -------
> 
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
> 
> 
> Thanks,
> 
> Bill Havanki
> 
>

Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory stress test

Posted by Bill Havanki <bh...@clouderagovt.com>.

(Updated July 1, 2014, 12:23 p.m.)

Review request for accumulo and Bill Slacum.

Changes
-------

Incorporated Mike's feedback.

Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947

Repository: accumulo

Description
-------

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/23093/diff/

Testing
-------

Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.

Thanks,

Bill Havanki