You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joey Frazee (Jira)" <ji...@apache.org> on 2020/11/10 22:46:00 UTC
[jira] [Commented] (NIFI-7964) PutAzureBlobStorage OutOfMemory
Exception
[ https://issues.apache.org/jira/browse/NIFI-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229554#comment-17229554 ]
Joey Frazee commented on NIFI-7964:
-----------------------------------
[~esecules] I forgot to mention, this was replicated on 1.9.x. I think it likely never worked but has just been hidden by heaps larger than the file size.
> PutAzureBlobStorage OutOfMemory Exception
> -----------------------------------------
>
> Key: NIFI-7964
> URL: https://issues.apache.org/jira/browse/NIFI-7964
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.12.1
> Reporter: Eric Secules
> Assignee: Joey Frazee
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of my flow I upload files to azure blob storage. They can be several hundred MB in size. I putting a 300 MB file into my flow and it choked on the PutAzureBlobStorage processor with the following log message.
> {code:java}
> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6] o.a.n.p.a.storage.PutAzureBlobStorage PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6] o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> {code}
> I did not expect this to happen because I think the PutAzureBlob processor should be streaming the flowfile from disk directly to blob. But this behaviour suggests to me that it's getting read into memory in its entirety. My JVM heap size is set to 512 MB, which shouldn't be a problem if streaming was used to upload to blob storage in chunks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)