[GitHub] [hadoop] goiri commented on a diff in pull request #4585: HADOOP-12007. GzipCodec native CodecPool leaks memory

[GitBox <gi...@apache.org>]

goiri commented on code in PR #4585:
URL: https://github.com/apache/hadoop/pull/4585#discussion_r924977086


##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecPool.java:
##########
@@ -189,4 +198,54 @@ public void testDecompressorNotReturnSameInstance() {
       CodecPool.returnDecompressor(decompressor);
     }
   }
+
+  @Test(timeout = 10000)
+  public void testDoNotPoolCompressorNotUseableAfterReturn() throws IOException {
+
+    final GzipCodec gzipCodec = new GzipCodec();
+    gzipCodec.setConf(new Configuration());
+
+    // BuiltInGzipCompressor is an explicit example of a Compressor with the @DoNotPool annotation
+    final Compressor compressor = new BuiltInGzipCompressor(new Configuration());
+    CodecPool.returnCompressor(compressor);
+
+    try (CompressionOutputStream outputStream =
+                 gzipCodec.createOutputStream(new ByteArrayOutputStream(), compressor)) {
+      outputStream.write(1);

Review Comment:
   LambdaTestUtils#intercept



##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecPool.java:
##########
@@ -189,4 +198,54 @@ public void testDecompressorNotReturnSameInstance() {
       CodecPool.returnDecompressor(decompressor);
     }
   }
+
+  @Test(timeout = 10000)
+  public void testDoNotPoolCompressorNotUseableAfterReturn() throws IOException {
+
+    final GzipCodec gzipCodec = new GzipCodec();
+    gzipCodec.setConf(new Configuration());
+
+    // BuiltInGzipCompressor is an explicit example of a Compressor with the @DoNotPool annotation
+    final Compressor compressor = new BuiltInGzipCompressor(new Configuration());
+    CodecPool.returnCompressor(compressor);
+
+    try (CompressionOutputStream outputStream =
+                 gzipCodec.createOutputStream(new ByteArrayOutputStream(), compressor)) {
+      outputStream.write(1);
+      fail("Compressor from Codec with @DoNotPool should not be useable after returning to CodecPool");
+    } catch (NullPointerException exception) {

Review Comment:
   NPE is the best we can do?



##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecPool.java:
##########
@@ -189,4 +198,54 @@ public void testDecompressorNotReturnSameInstance() {
       CodecPool.returnDecompressor(decompressor);
     }
   }
+
+  @Test(timeout = 10000)
+  public void testDoNotPoolCompressorNotUseableAfterReturn() throws IOException {
+
+    final GzipCodec gzipCodec = new GzipCodec();
+    gzipCodec.setConf(new Configuration());
+
+    // BuiltInGzipCompressor is an explicit example of a Compressor with the @DoNotPool annotation
+    final Compressor compressor = new BuiltInGzipCompressor(new Configuration());
+    CodecPool.returnCompressor(compressor);
+
+    try (CompressionOutputStream outputStream =
+                 gzipCodec.createOutputStream(new ByteArrayOutputStream(), compressor)) {
+      outputStream.write(1);
+      fail("Compressor from Codec with @DoNotPool should not be useable after returning to CodecPool");
+    } catch (NullPointerException exception) {
+      Assert.assertEquals("Deflater has been closed", exception.getMessage());
+    }
+  }
+
+  @Test(timeout = 10000)
+  public void testDoNotPoolDecompressorNotUseableAfterReturn() throws IOException {
+
+    final GzipCodec gzipCodec = new GzipCodec();
+    gzipCodec.setConf(new Configuration());
+
+    final Random random = new Random();
+    final byte[] bytes = new byte[1024];
+    random.nextBytes(bytes);
+
+    ByteArrayOutputStream baos = new ByteArrayOutputStream();
+    try (OutputStream outputStream = gzipCodec.createOutputStream(baos)) {
+      outputStream.write(bytes);
+    }
+
+    final byte[] gzipBytes = baos.toByteArray();
+    final ByteArrayInputStream bais = new ByteArrayInputStream(gzipBytes);
+
+    // BuiltInGzipDecompressor is an explicit example of a Decompressor with the @DoNotPool annotation
+    final Decompressor decompressor = new BuiltInGzipDecompressor();
+    CodecPool.returnDecompressor(decompressor);
+
+    try (CompressionInputStream inputStream =

Review Comment:
   ```
       try (CompressionInputStream inputStream = gzipCodec.createInputStream(bais, decompressor)) {
         LambdaTestUtils.intercept(
             NullPointerException.class,
             "Decompressor from Codec with @DoNotPool should not be useable after returning to CodecPool"
             () -> inputStream.read());
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org