You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Alex Herbert (Jira)" <ji...@apache.org> on 2019/12/02 22:51:00 UTC
[jira] [Commented] (CODEC-265)
java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/CODEC-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986436#comment-16986436 ]
Alex Herbert commented on CODEC-265:
------------------------------------
Smarter memory allocation has been implemented for the Base32/Base64 codecs. They can now allocate up to the system limit for arrays.
Please check against master. [Snapshots|https://repository.apache.org/content/repositories/snapshots/commons-codec/commons-codec/1.14-SNAPSHOT/] are now deployed so you can include the 1.14 snapshot dependency for testing.
In the source there is a new test Base64Test.testCodec265 that demonstrates encoding a 1 gigabyte array. The test is memory hungry.
In a real world use case it may be worth looking at the streaming API using Base64OutputStream to write chunks of byte[] data to your target output. Even if this is a ByteArrayOutputStream you will have lower memory usage as you will not have to read the entire allocated random file to memory and you can convert the ByteArrayOutputStream to a String directly. Memory usage will still peak at around 4GiB as the String will have 2 bytes for each character:
{code:java}
String filePath = "/tmp/1gb.zip";
Path path = Paths.get(filePath);
String contentToBeSaved;
try (InputStream in = Files.newInputStream(path)) {
// Allocate 1.5 GiB to avoid buffer reallocation
final ByteArrayOutputStream out = new ByteArrayOutputStream(3 * (1 << 29));
// Encode with no line separator
final Base64OutputStream base64 = new Base64OutputStream(out, true, 0, null);
final byte[] buffer = new byte[8192];
int read = in.read(buffer);
while (read >= 0) {
base64.write(buffer, 0, read);
read = in.read(buffer);
}
base64.close();
// This will require 2.67GiB ~ 2 * 1GiB * 4/3
contentToBeSaved = out.toString();
}
{code}
There will be a fair bit more memory usage for the JSONObject in your example. Hopefully codec is not a blocker any more and you can continue your testing.
> java.lang.NegativeArraySizeException
> --------------------------------------
>
> Key: CODEC-265
> URL: https://issues.apache.org/jira/browse/CODEC-265
> Project: Commons Codec
> Issue Type: Bug
> Affects Versions: 1.13
> Environment: Linux = Ubuntu 18.04.3 LTS
> JDK = 1.8
>
> Reporter: Ingimar
> Assignee: Alex Herbert
> Priority: Critical
> Attachments: NewClientEncodePost.java, Util.java, pom.xml
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi,
> trying to encode a file that is 1GB of size.
> ( linux :
> {code:java}
> fallocate -l 1GB 1gb.zip{code}
> )
> I want to post that file to a RESTful-service, package in JSON.
> *here is the code*
>
>
> {code:java}
> String filePath = "/tmp/1gb.zip";
> System.out.println("\t Post to : ".concat(URL));
> System.out.println("\t file : ".concat(filePath));
> Path path = Paths.get(filePath);
> byte[] bArray = Files.readAllBytes(path);
> // testing commons codec 1.16 (2019-11-05)
> byte[] encodeBase64 = Base64.encodeBase64(bArray);
> final String contentToBeSaved = new String(encodeBase64);
> HttpClient client = HttpClientBuilder.create().build();
> HttpResponse response = null;
> JSONObject metadata = new JSONObject();
> metadata.put("owner", "Ingo");
> metadata.put("access", "public");
> metadata.put("licenseType", "CC BY");
> metadata.put("fileName", "fileName");
> metadata.put("fileDataBase64", contentToBeSaved);
> String metadataFormatted = StringEscapeUtils.unescapeJavaScript(metadata.toString());
> StringEntity entity = new StringEntity(metadataFormatted, ContentType.APPLICATION_JSON);
> HttpPost post = new HttpPost(URL);
> post.setEntity(entity);
> response = client.execute(post);
> HttpEntity responseEntity = response.getEntity();
> String responseFromMediaserver = EntityUtils.toString(responseEntity, "UTF-8");
> System.out.println("\n****");
> System.out.println("Response is : " + responseFromMediaserver);
> JSONObject json = new JSONObject(responseFromMediaserver);
> String uuid = json.getString("uuid");
> System.out.println("UUID is " + uuid);
> {code}
>
>
> # mvn clean package
> # java -Xms512m -Xmx20480m -jar target/mediaClient.jar
> The crasch is in
>
> {code:java}
> byte[] encodeBase64 = Base64.encodeBase64(bArray);{code}
>
> the stacktrace is :
> {code:java}
>
> Starting NewClientEncodePost
> Post to : http://127.0.0.1:8080/MediaServerResteasy/media
> file : /tmp/1gb.zip
> Exception in thread "main" java.lang.NegativeArraySizeException
> at org.apache.commons.codec.binary.BaseNCodec.resizeBuffer(BaseNCodec.java:253)
> at org.apache.commons.codec.binary.BaseNCodec.ensureBufferSize(BaseNCodec.java:269)
> at org.apache.commons.codec.binary.Base64.encode(Base64.java:380)
> at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:451)
> at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:430)
> at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:679)
> at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:642)
> at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:623)
> at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:556)
> at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.posting(NewClientEncodePost.java:55)
> at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.main(NewClientEncodePost.java:38)
>
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)