You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/01/06 12:49:00 UTC
[jira] [Commented] (NIFI-7964) PutAzureBlobStorage OutOfMemory
Exception
[ https://issues.apache.org/jira/browse/NIFI-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259681#comment-17259681 ]
ASF subversion and git services commented on NIFI-7964:
-------------------------------------------------------
Commit 1e227ca6436899b8109a3443aa4487bd43146fca in nifi's branch refs/heads/main from Joey Frazee
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1e227ca ]
NIFI-7964 Force PutAzureBlobStorage to stream writes
NIFI-7964 Remove unused import
NIFI-7964 Reverted to using BufferedInputStream along with unmarkable
NIFI-7964 Made UnmarkableInputStream static and added factory method
NIFI-7964 Remove unnecessary factory method from UnmarkableInputStream
This closes #4632.
Signed-off-by: Peter Turcsanyi <tu...@apache.org>
> PutAzureBlobStorage OutOfMemory Exception
> -----------------------------------------
>
> Key: NIFI-7964
> URL: https://issues.apache.org/jira/browse/NIFI-7964
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.12.1
> Reporter: Eric Secules
> Assignee: Joey Frazee
> Priority: Blocker
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of my flow I upload files to azure blob storage. They can be several hundred MB in size. I putting a 300 MB file into my flow and it choked on the PutAzureBlobStorage processor with the following log message.
> {code:java}
> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6] o.a.n.p.a.storage.PutAzureBlobStorage PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6] o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to uncaught Exception: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> {code}
> I did not expect this to happen because I think the PutAzureBlob processor should be streaming the flowfile from disk directly to blob. But this behaviour suggests to me that it's getting read into memory in its entirety. My JVM heap size is set to 512 MB, which shouldn't be a problem if streaming was used to upload to blob storage in chunks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)