You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Kevin Sweeney <ke...@apache.org> on 2014/09/25 03:35:24 UTC

Review Request 26012: Use BufferedOutputStream for deflater

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

Review request for Aurora, David McLaughlin and Bill Farner.


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing
-------

./gradlew build

Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯

# Deflater.BEST_SPEED, No buffer
deflate: 11745ms
total:12734ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, buf = 512KiB
deflate: 11015ms
total:12001ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream
deflate: 6885ms
total:7694ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
deflate: 6752ms
total:7585ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.DEFAULT_COMPRESSION
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
deflate: 16516ms
total:17103ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
deflate: 12241ms
total:12788ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
deflate: 11548ms
total:12108ms
compression ratio: 647270930/92690409 = 698.31%


Thanks,

Kevin Sweeney

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Kevin Sweeney <ke...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

(Updated Sept. 25, 2014, 3:21 p.m.)


Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.


Changes
-------

Fix truncated line


Bugs: AURORA-722
    https://issues.apache.org/jira/browse/AURORA-722


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing (updated)
-------

./gradlew build

Microbenchmark with a scheduler backup from my production environment.

Before:
## Deflater.DEFAULT_COMPRESSION, (No BufferedOutputStream), DeflaterOutputStream(512B)
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

After:
# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream(256KiB), DeflaterOutputStream(256KiB)
deflate: 11807ms (4857ms speedup)
total: 12379ms (4840ms speedup)
compression ratio: 647270930/92690409 = 698.31% (0% improvement)


Thanks,

Kevin Sweeney

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Kevin Sweeney <ke...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

(Updated Sept. 25, 2014, 3:12 p.m.)


Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.


Changes
-------

Update Testing Done to reflect current code


Bugs: AURORA-722
    https://issues.apache.org/jira/browse/AURORA-722


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs (updated)
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing (updated)
-------

./gradlew build

Microbenchmark with a scheduler backup from my production environment.

Before:
## Deflater.DEFAULT_COMPRESSION, (No BufferedOutputStream), DeflaterOutputStream(
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

After:
# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream(256KiB), DeflaterOutputStream(256KiB)
deflate: 11807ms (4857ms speedup)
total: 12379ms (4840ms speedup)
compression ratio: 647270930/92690409 = 698.31% (0% improvement)


Thanks,

Kevin Sweeney

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Kevin Sweeney <ke...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

(Updated Sept. 25, 2014, 2:24 p.m.)


Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.


Changes
-------

Add bug link.


Bugs: AURORA-722
    https://issues.apache.org/jira/browse/AURORA-722


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing
-------

./gradlew build

Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯

# Deflater.BEST_SPEED, No buffer
deflate: 11745ms
total:12734ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, buf = 512KiB
deflate: 11015ms
total:12001ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream
deflate: 6885ms
total:7694ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
deflate: 6752ms
total:7585ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.DEFAULT_COMPRESSION
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
deflate: 16516ms
total:17103ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
deflate: 12241ms
total:12788ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
deflate: 11548ms
total:12108ms
compression ratio: 647270930/92690409 = 698.31%


Thanks,

Kevin Sweeney

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Kevin Sweeney <ke...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

(Updated Sept. 25, 2014, 2:24 p.m.)


Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.


Changes
-------

Add explanatory comment.


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs (updated)
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing
-------

./gradlew build

Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯

# Deflater.BEST_SPEED, No buffer
deflate: 11745ms
total:12734ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, buf = 512KiB
deflate: 11015ms
total:12001ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream
deflate: 6885ms
total:7694ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
deflate: 6752ms
total:7585ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.DEFAULT_COMPRESSION
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
deflate: 16516ms
total:17103ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
deflate: 12241ms
total:12788ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
deflate: 11548ms
total:12108ms
compression ratio: 647270930/92690409 = 698.31%


Thanks,

Kevin Sweeney

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Bill Farner <wf...@apache.org>.

> On Sept. 25, 2014, 2:23 a.m., Bill Farner wrote:
> > Ship It!

Actually - can you include a comment with a very brief explanation of the findings?  Wouldn't want somebody to remove the seemingly-unnecessary stream wrapper.


- Bill


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/#review54500
-----------------------------------------------------------


On Sept. 25, 2014, 1:36 a.m., Kevin Sweeney wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26012/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2014, 1:36 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Use BufferedOutputStream for deflater
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 
> 
> Diff: https://reviews.apache.org/r/26012/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew build
> 
> Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯
> 
> # Deflater.BEST_SPEED, No buffer
> deflate: 11745ms
> total:12734ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, buf = 512KiB
> deflate: 11015ms
> total:12001ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream
> deflate: 6885ms
> total:7694ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
> deflate: 6752ms
> total:7585ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.DEFAULT_COMPRESSION
> deflate: 16664ms
> total:17219ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
> deflate: 16516ms
> total:17103ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
> deflate: 12241ms
> total:12788ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
> deflate: 11548ms
> total:12108ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> 
> Thanks,
> 
> Kevin Sweeney
> 
>

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Bill Farner <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/#review54500
-----------------------------------------------------------

Ship it!


Ship It!

- Bill Farner


On Sept. 25, 2014, 1:36 a.m., Kevin Sweeney wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26012/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2014, 1:36 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Use BufferedOutputStream for deflater
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 
> 
> Diff: https://reviews.apache.org/r/26012/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew build
> 
> Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯
> 
> # Deflater.BEST_SPEED, No buffer
> deflate: 11745ms
> total:12734ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, buf = 512KiB
> deflate: 11015ms
> total:12001ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream
> deflate: 6885ms
> total:7694ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
> deflate: 6752ms
> total:7585ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.DEFAULT_COMPRESSION
> deflate: 16664ms
> total:17219ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
> deflate: 16516ms
> total:17103ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
> deflate: 12241ms
> total:12788ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
> deflate: 11548ms
> total:12108ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> 
> Thanks,
> 
> Kevin Sweeney
> 
>

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by David McLaughlin <da...@dmclaughlin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/#review54494
-----------------------------------------------------------

Ship it!


Ship It!

- David McLaughlin


On Sept. 25, 2014, 1:36 a.m., Kevin Sweeney wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26012/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2014, 1:36 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Use BufferedOutputStream for deflater
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 
> 
> Diff: https://reviews.apache.org/r/26012/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew build
> 
> Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯
> 
> # Deflater.BEST_SPEED, No buffer
> deflate: 11745ms
> total:12734ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, buf = 512KiB
> deflate: 11015ms
> total:12001ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream
> deflate: 6885ms
> total:7694ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
> deflate: 6752ms
> total:7585ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.DEFAULT_COMPRESSION
> deflate: 16664ms
> total:17219ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
> deflate: 16516ms
> total:17103ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
> deflate: 12241ms
> total:12788ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
> deflate: 11548ms
> total:12108ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> 
> Thanks,
> 
> Kevin Sweeney
> 
>

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/#review54492
-----------------------------------------------------------

Ship it!


Ship It!

- Maxim Khutornenko


On Sept. 25, 2014, 1:36 a.m., Kevin Sweeney wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26012/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2014, 1:36 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Use BufferedOutputStream for deflater
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 
> 
> Diff: https://reviews.apache.org/r/26012/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew build
> 
> Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯
> 
> # Deflater.BEST_SPEED, No buffer
> deflate: 11745ms
> total:12734ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, buf = 512KiB
> deflate: 11015ms
> total:12001ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream
> deflate: 6885ms
> total:7694ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
> deflate: 6752ms
> total:7585ms
> compression ratio: 647270930/121616834 = 532.22%
> 
> # Deflater.DEFAULT_COMPRESSION
> deflate: 16664ms
> total:17219ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
> deflate: 16516ms
> total:17103ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
> deflate: 12241ms
> total:12788ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> # Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
> deflate: 11548ms
> total:12108ms
> compression ratio: 647270930/92690409 = 698.31%
> 
> 
> Thanks,
> 
> Kevin Sweeney
> 
>

Re: Review Request 26012: Use BufferedOutputStream for deflater

Posted by Kevin Sweeney <ke...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26012/
-----------------------------------------------------------

(Updated Sept. 24, 2014, 6:36 p.m.)


Review request for Aurora, David McLaughlin, Joe Smith, and Bill Farner.


Changes
-------

+jsmith


Repository: aurora


Description
-------

Use BufferedOutputStream for deflater


Diffs
-----

  src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 45cf7ecf52582ba3fd7bbc8b4a981e396793da4b 

Diff: https://reviews.apache.org/r/26012/diff/


Testing
-------

./gradlew build

Also put together a microbenchmark with a scheduler backup from my production environment - using a BufferedOutputStream gets a 4sec speedup with no reduction in compression ratio. ¯_(?)_/¯

# Deflater.BEST_SPEED, No buffer
deflate: 11745ms
total:12734ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, buf = 512KiB
deflate: 11015ms
total:12001ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream
deflate: 6885ms
total:7694ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.BEST_SPEED, BufferedOutputStream at 512KiB
deflate: 6752ms
total:7585ms
compression ratio: 647270930/121616834 = 532.22%

# Deflater.DEFAULT_COMPRESSION
deflate: 16664ms
total:17219ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, buf size = 512KiB
deflate: 16516ms
total:17103ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream
deflate: 12241ms
total:12788ms
compression ratio: 647270930/92690409 = 698.31%

# Deflater.DEFAULT_COMPRESSION, BufferedOutputStream at 512KiB
deflate: 11548ms
total:12108ms
compression ratio: 647270930/92690409 = 698.31%


Thanks,

Kevin Sweeney