[jira] [Created] (NIFI-7964) PutAzureBlobStorage OutOfMemory Exception

["Eric Secules (Jira)" <ji...@apache.org>]

Eric Secules created NIFI-7964:
----------------------------------

             Summary: PutAzureBlobStorage OutOfMemory Exception
                 Key: NIFI-7964
                 URL: https://issues.apache.org/jira/browse/NIFI-7964
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.12.1
            Reporter: Eric Secules


As part of my flow I upload files to azure blob storage. They can be several hundred MB in size. I putting a 300 MB file into my flow and it choked on the PutAzureBlobStorage processor with the following log message.
{code:java}
2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6] o.a.n.p.a.storage.PutAzureBlobStorage PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6] o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
{code}
I did not expect this to happen because I think the PutAzureBlob processor should be streaming the flowfile from disk directly to blob. But this behaviour suggests to me that it's getting read into memory in its entirety. My JVM heap size is set to 512 MB, which shouldn't be a problem if streaming was used to upload to blob storage in chunks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)