You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Bill Havanki <bh...@clouderagovt.com> on 2014/07/01 18:23:33 UTC
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/
-----------------------------------------------------------
(Updated July 1, 2014, 12:23 p.m.)
Review request for accumulo and Bill Slacum.
Changes
-------
Incorporated Mike's feedback.
Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947
Repository: accumulo
Description
-------
A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
Diffs (updated)
-----
test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
test/system/stress/stress-env.sh.example 1360c67
test/system/stress/writer.sh 7d9b283
Diff: https://reviews.apache.org/r/23093/diff/
Testing
-------
Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
Thanks,
Bill Havanki
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Josh Elser <jo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47128
-----------------------------------------------------------
Ship it!
Ship It!
- Josh Elser
On July 1, 2014, 7 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 7 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Bill Havanki <bh...@clouderagovt.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/
-----------------------------------------------------------
(Updated July 1, 2014, 3 p.m.)
Review request for accumulo and Bill Slacum.
Changes
-------
Incorporated Josh's feedback.
Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947
Repository: accumulo
Description
-------
A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
Diffs (updated)
-----
test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
test/system/stress/stress-env.sh.example 1360c67
test/system/stress/writer.sh 7d9b283
Diff: https://reviews.apache.org/r/23093/diff/
Testing
-------
Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
Thanks,
Bill Havanki
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Bill Havanki <bh...@clouderagovt.com>.
> On July 1, 2014, 1:43 p.m., Josh Elser wrote:
> > test/system/stress/stress-env.sh.example, line 55
> > <https://reviews.apache.org/r/23093/diff/3/?file=621403#file621403line55>
> >
> > Would be nice to include what the default -1 value equates to in the comment (in addition to the help message) since you have a nice comment describing the argument already.
Good deal.
- Bill
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47105
-----------------------------------------------------------
On July 1, 2014, 3 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 3 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Josh Elser <jo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47105
-----------------------------------------------------------
test/system/stress/stress-env.sh.example
<https://reviews.apache.org/r/23093/#comment82710>
Would be nice to include what the default -1 value equates to in the comment (in addition to the help message) since you have a nice comment describing the argument already.
- Josh Elser
On July 1, 2014, 5:09 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 5:09 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Bill Havanki <bh...@clouderagovt.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/
-----------------------------------------------------------
(Updated July 1, 2014, 1:09 p.m.)
Review request for accumulo and Bill Slacum.
Changes
-------
More of Mike's feedback.
Bugs: ACCUMULO-2947
https://issues.apache.org/jira/browse/ACCUMULO-2947
Repository: accumulo
Description
-------
A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
Diffs (updated)
-----
test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
test/system/stress/stress-env.sh.example 1360c67
test/system/stress/writer.sh 7d9b283
Diff: https://reviews.apache.org/r/23093/diff/
Testing
-------
Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
Thanks,
Bill Havanki
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Mike Drob <md...@mdrob.com>.
> On July 1, 2014, 4:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 53
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line53>
> >
> > I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.
>
> Bill Havanki wrote:
> I hadn't thought of that, actually ... it just turned out that way. At this stage, I would be fine with it failing horribly if a put() throws an exception. In that case, I can actually just say this once:
>
> cells_remaining_in_row -= cells;
>
> How's that?
Sure, as long as you do it after the loop.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------
On July 1, 2014, 4:23 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 4:23 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Bill Havanki <bh...@clouderagovt.com>.
> On July 1, 2014, 12:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 50
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line50>
> >
> > I don't know why, but I like this. I would have done (i < max && i < remaining), but this is much cleaner. :)
I did have something like (i < max && i < remaining) at first, but realized the whole thing could be precomputed since max and remaining are invariant.
> On July 1, 2014, 12:29 p.m., Mike Drob wrote:
> > test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java, line 53
> > <https://reviews.apache.org/r/23093/diff/2/?file=621369#file621369line53>
> >
> > I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.
I hadn't thought of that, actually ... it just turned out that way. At this stage, I would be fine with it failing horribly if a put() throws an exception. In that case, I can actually just say this once:
cells_remaining_in_row -= cells;
How's that?
- Bill
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------
On July 1, 2014, 12:23 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 12:23 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>
Re: Review Request 23093: ACCUMULO-2947 - Limit cells per mutation in memory
stress test
Posted by Mike Drob <md...@mdrob.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23093/#review47078
-----------------------------------------------------------
test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java
<https://reviews.apache.org/r/23093/#comment82681>
I don't know why, but I like this. I would have done (i < max && i < remaining), but this is much cleaner. :)
test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java
<https://reviews.apache.org/r/23093/#comment82679>
I assume we are decrementing this after each put() instead of all at once is because we are worried about an exception getting thrown? In which case, it would make sense to not decrement the remaining count at all, since those mutation bits will never be returned.
- Mike Drob
On July 1, 2014, 4:23 p.m., Bill Havanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23093/
> -----------------------------------------------------------
>
> (Updated July 1, 2014, 4:23 p.m.)
>
>
> Review request for accumulo and Bill Slacum.
>
>
> Bugs: ACCUMULO-2947
> https://issues.apache.org/jira/browse/ACCUMULO-2947
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> A new switch to the memory stress test Write utility, --max-cells-per-mutation, controls the maximum number of cells to be written in any mutation. If the maximum is hit before the configured row width, the current mutation is used and the next mutation picks up where it left off, in the same row.
>
> The core logic is in the RandomMutations class. It uses a counter for how many cells are remaining in the current row, with 0 indicating that a new row should be started. Otherwise, the last row is used. The counter is used as the loop counter for writing cells. When it hits zero or the maximum, writes stop.
>
>
> Diffs
> -----
>
> test/src/main/java/org/apache/accumulo/test/stress/random/RandomMutations.java c4504b2
> test/src/main/java/org/apache/accumulo/test/stress/random/Write.java 9c29871
> test/src/main/java/org/apache/accumulo/test/stress/random/WriteOptions.java c213528
> test/system/stress/stress-env.sh.example 1360c67
> test/system/stress/writer.sh 7d9b283
>
> Diff: https://reviews.apache.org/r/23093/diff/
>
>
> Testing
> -------
>
> Ran memory stress test on single-node 1.5.2 cluster. With max-cells-per-mutation = 4 and row-width ranging from 1 to 7, saw rows still being written with 5 or more cells.
>
>
> Thanks,
>
> Bill Havanki
>
>