You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by John McNamee <jp...@terpmail.umd.edu> on 2014/05/22 21:57:28 UTC
Review Request 21826: Accumulo-2632
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/
-----------------------------------------------------------
Review request for accumulo.
Repository: accumulo
Description
-------
The goal was to find a way to increase the write speed in the DfsLogger using compression.
This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
The option to use compression has been made configurable.
Recovery has been modified to uncompress the WAL if needed.
Diffs
-----
core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/21826/diff/
Testing
-------
There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
Thanks,
John McNamee
Re: Review Request 21826: Accumulo-2632
Posted by John McNamee <jp...@terpmail.umd.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/
-----------------------------------------------------------
(Updated Nov. 24, 2014, 1:26 p.m.)
Review request for accumulo.
Changes
-------
Link to the JIRA issue.
Bugs: ACCUMULO-2632
https://issues.apache.org/jira/browse/ACCUMULO-2632
Repository: accumulo
Description
-------
The goal was to find a way to increase the write speed in the DfsLogger using compression.
This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
The option to use compression has been made configurable.
Recovery has been modified to uncompress the WAL if needed.
Diffs
-----
core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/21826/diff/
Testing
-------
There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
Thanks,
John McNamee
Re: Review Request 21826: Accumulo-2632
Posted by Christopher Tubbs <ct...@apache.org>.
> On May 22, 2014, 4:28 p.m., Josh Elser wrote:
> > core/src/main/java/org/apache/accumulo/core/conf/Property.java, line 270
> > <https://reviews.apache.org/r/21826/diff/1/?file=588464#file588464line270>
> >
> > Should this be tagged with the @Experimental annotation?
I view @Experimental to denote features that are more invasive, optional, and may not work as expected, not simply to denote things that users can experiment with. @Experimental hides these configuration items from a typical user's view, making the target audience developers, rather than end users. While it is currently experimental as a patch, I think the expectation here is that when this feature is ready for merging into a branch, the feature will be targeted to end users and made available for them to tweak, so @Experimental doesn't quite apply.
> On May 22, 2014, 4:28 p.m., Josh Elser wrote:
> > server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java, line 241
> > <https://reviews.apache.org/r/21826/diff/1/?file=588466#file588466line241>
> >
> > Could we leverage the Compression stuff in Hadoop rather than invoke the snappy lib directly?
I agree, this feature could be improved by allowing more compression types in the configuration, rather than simply enable/disable, and that should probably be done before merging in. However, there are special considerations with other compression types. Snappy allows you to take a byte array and make another byte array. The other types work with streams, and you'd have to ensure the stream is flushed before sync is called. Snappy's byte[]-to-byte[] is a bit easier to experiment with, as a first pass.
- Christopher
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/#review43764
-----------------------------------------------------------
On May 22, 2014, 3:57 p.m., John McNamee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21826/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 3:57 p.m.)
>
>
> Review request for accumulo.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> The goal was to find a way to increase the write speed in the DfsLogger using compression.
> This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
> The option to use compression has been made configurable.
> Recovery has been modified to uncompress the WAL if needed.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
> server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/21826/diff/
>
>
> Testing
> -------
>
> There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
>
>
> Thanks,
>
> John McNamee
>
>
Re: Review Request 21826: Accumulo-2632
Posted by Christopher Tubbs <ct...@apache.org>.
> On May 22, 2014, 4:28 p.m., Josh Elser wrote:
> > server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java, line 572
> > <https://reviews.apache.org/r/21826/diff/1/?file=588466#file588466line572>
> >
> > It looks like this sync invocation got dropped. Did I just miss it moving somewhere?
I think Keith was working on ACCUMULO-2766. Any feature improvement here should probably be rebased on top of that, to ensure that the concurrency issues are correct.
- Christopher
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/#review43764
-----------------------------------------------------------
On May 22, 2014, 3:57 p.m., John McNamee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21826/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 3:57 p.m.)
>
>
> Review request for accumulo.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> The goal was to find a way to increase the write speed in the DfsLogger using compression.
> This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
> The option to use compression has been made configurable.
> Recovery has been modified to uncompress the WAL if needed.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
> server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/21826/diff/
>
>
> Testing
> -------
>
> There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
>
>
> Thanks,
>
> John McNamee
>
>
Re: Review Request 21826: Accumulo-2632
Posted by Josh Elser <jo...@gmail.com>.
> On May 22, 2014, 8:28 p.m., Josh Elser wrote:
> > Have you done any benchmarks with these changes? I would think that you should see higher CPU usage with less disk I/O, and, overall, quicker ingest. It would be good to see some before and after numbers that show this.
Oops, sorry. I didn't see the attachment you put on the ticket :)
- Josh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/#review43764
-----------------------------------------------------------
On May 22, 2014, 7:57 p.m., John McNamee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21826/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 7:57 p.m.)
>
>
> Review request for accumulo.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> The goal was to find a way to increase the write speed in the DfsLogger using compression.
> This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
> The option to use compression has been made configurable.
> Recovery has been modified to uncompress the WAL if needed.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
> server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/21826/diff/
>
>
> Testing
> -------
>
> There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
>
>
> Thanks,
>
> John McNamee
>
>
Re: Review Request 21826: Accumulo-2632
Posted by Josh Elser <jo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21826/#review43764
-----------------------------------------------------------
Have you done any benchmarks with these changes? I would think that you should see higher CPU usage with less disk I/O, and, overall, quicker ingest. It would be good to see some before and after numbers that show this.
core/src/main/java/org/apache/accumulo/core/conf/Property.java
<https://reviews.apache.org/r/21826/#comment78049>
Should this be tagged with the @Experimental annotation?
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java
<https://reviews.apache.org/r/21826/#comment78054>
Same comment as elsewhere, can we use what Hadoop has for compression to abstract out the specifics of Snappy?
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java
<https://reviews.apache.org/r/21826/#comment78051>
Could we leverage the Compression stuff in Hadoop rather than invoke the snappy lib directly?
server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java
<https://reviews.apache.org/r/21826/#comment78058>
It looks like this sync invocation got dropped. Did I just miss it moving somewhere?
- Josh Elser
On May 22, 2014, 7:57 p.m., John McNamee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21826/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 7:57 p.m.)
>
>
> Review request for accumulo.
>
>
> Repository: accumulo
>
>
> Description
> -------
>
> The goal was to find a way to increase the write speed in the DfsLogger using compression.
> This patch uses the Snappy compression library and was the most optimum way that I could find to increase write performance.
> The option to use compression has been made configurable.
> Recovery has been modified to uncompress the WAL if needed.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DecompressingInputStream.java PRE-CREATION
> server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java bfa51d3
> server/tserver/src/test/java/org/apache/accumulo/tserver/log/DecompressingInputStreamTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/21826/diff/
>
>
> Testing
> -------
>
> There is a unit test for the DecryptingInputStream.java, and I ran continuous Ingest/ continuous verify with the agitator on.
>
>
> Thanks,
>
> John McNamee
>
>