[jira] [Created] (SANTUARIO-459) Wrong usage of Integer.MAX_VALUE

["Ronald Boettcher (JIRA)" <ji...@apache.org>]

Ronald Boettcher created SANTUARIO-459:
------------------------------------------

             Summary: Wrong usage of Integer.MAX_VALUE
                 Key: SANTUARIO-459
                 URL: https://issues.apache.org/jira/browse/SANTUARIO-459
             Project: Santuario
          Issue Type: Bug
         Environment: Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64

            Reporter: Ronald Boettcher
            Assignee: Colm O hEigeartaigh
            Priority: Minor


Most Java VMs are throwing a "Requested array size exceeds VM limit" OutOfMemoryException when using just Integer.MAX_VALUE or Integer.MAX_VALUE - 1

We are using Apache Santuario XML Encryption with large documents and encountered this problem.

I had to change it in org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java.

The patch file content is as following (I'm missing an attachment button):

{code:title=UnsyncByteArrayOutputStream.java.patch|borderStyle=solid}
*** src/main/java/org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java.orig   2016-09-30 11:59:49.570187935 +0200
--- src/main/java/org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java        2016-09-30 11:59:31.149998245 +0200
***************
*** 28,33 ****
--- 28,37 ----
  public class UnsyncByteArrayOutputStream extends OutputStream  {
  
      private static final int INITIAL_SIZE = 8192;
+     
+     // most Java VMs are throwing a "Requested array size exceeds VM limit" OutOfMemoryException
+     // when using just Integer.MAX_VALUE or Integer.MAX_VALUE - 1
+     private static final int VM_ARRAY_INDEX_MAX_VALUE = Integer.MAX_VALUE - 2;
  
      private byte[] buf;
      private int size = INITIAL_SIZE;
***************
*** 38,44 ****
      }
  
      public void write(byte[] arg0) {
!         if ((Integer.MAX_VALUE - pos) < arg0.length) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg0.length;
--- 42,48 ----
      }
  
      public void write(byte[] arg0) {
!         if ((VM_ARRAY_INDEX_MAX_VALUE - pos) < arg0.length) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg0.length;
***************
*** 50,56 ****
      }
  
      public void write(byte[] arg0, int arg1, int arg2) {
!         if ((Integer.MAX_VALUE - pos) < arg2) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg2;
--- 54,60 ----
      }
  
      public void write(byte[] arg0, int arg1, int arg2) {
!         if ((VM_ARRAY_INDEX_MAX_VALUE - pos) < arg2) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg2;
***************
*** 62,68 ****
      }
  
      public void write(int arg0) {
!         if (Integer.MAX_VALUE - pos == 0) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + 1;
--- 66,72 ----
      }
  
      public void write(int arg0) {
!         if (VM_ARRAY_INDEX_MAX_VALUE - pos == 0) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + 1;
***************
*** 88,94 ****
              newSize = newSize << 1;
              // Deal with overflow
              if (newSize < 0) {
!                 newSize = Integer.MAX_VALUE;
              }
          }
          byte newBuf[] = new byte[newSize];
--- 92,98 ----
              newSize = newSize << 1;
              // Deal with overflow
              if (newSize < 0) {
!                 newSize = VM_ARRAY_INDEX_MAX_VALUE;
              }
          }
          byte newBuf[] = new byte[newSize];
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)