You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Eric Secules <es...@gmail.com> on 2020/10/28 19:40:23 UTC
PutAzureBlobStorage OutOfMemoryError
Hello everyone,
I am trying to upload a 300 MB file to azure blob storage using
PutAzureBlobStorage and the processor is failing due to an
OutOfMemoryError. My JVM heap size is set to 512 MB, but I wouldn't expect
this to be an issue because the PutAzureBlobStorage processor should be
using streaming to send the file to azure in chunks rather than reading it
into memory in its entirety and then sending it out.
I am using NiFi version 1.12.1
Here's the error from the NiFi logs:
> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6]
> o.a.n.p.a.storage.PutAzureBlobStorage
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5]
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to
> process session due to java.lang.OutOfMemoryError: Java heap space;
> Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError:
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6]
> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to
> uncaught Exception: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>
What's the recommendation for using NiFi to upload files of this size to
blob storage?
Thanks,
Eric
Re: PutAzureBlobStorage OutOfMemoryError
Posted by Eric Secules <es...@gmail.com>.
Hi Joe,
Thanks for confirming, I logged this bug
https://issues.apache.org/jira/browse/NIFI-7964
-Eric
On Wed, Oct 28, 2020 at 2:10 PM Joe Witt <jo...@gmail.com> wrote:
> Eric
>
> Your assumption is definitely what it should be. This will need someone
> to investigate. Please file a jira with as much detail as you can.
>
> Thanks
>
> On Wed, Oct 28, 2020 at 12:40 PM Eric Secules <es...@gmail.com> wrote:
>
>> Hello everyone,
>>
>> I am trying to upload a 300 MB file to azure blob storage using
>> PutAzureBlobStorage and the processor is failing due to an
>> OutOfMemoryError. My JVM heap size is set to 512 MB, but I wouldn't expect
>> this to be an issue because the PutAzureBlobStorage processor should be
>> using streaming to send the file to azure in chunks rather than reading it
>> into memory in its entirety and then sending it out.
>>
>> I am using NiFi version 1.12.1
>>
>> Here's the error from the NiFi logs:
>>
>>> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6]
>>> o.a.n.p.a.storage.PutAzureBlobStorage
>>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5]
>>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to
>>> process session due to java.lang.OutOfMemoryError: Java heap space;
>>> Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError:
>>> Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6]
>>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
>>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to
>>> uncaught Exception: java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>
>> What's the recommendation for using NiFi to upload files of this size to
>> blob storage?
>>
>> Thanks,
>> Eric
>>
>
Re: PutAzureBlobStorage OutOfMemoryError
Posted by Joe Witt <jo...@gmail.com>.
Eric
Your assumption is definitely what it should be. This will need someone to
investigate. Please file a jira with as much detail as you can.
Thanks
On Wed, Oct 28, 2020 at 12:40 PM Eric Secules <es...@gmail.com> wrote:
> Hello everyone,
>
> I am trying to upload a 300 MB file to azure blob storage using
> PutAzureBlobStorage and the processor is failing due to an
> OutOfMemoryError. My JVM heap size is set to 512 MB, but I wouldn't expect
> this to be an issue because the PutAzureBlobStorage processor should be
> using streaming to send the file to azure in chunks rather than reading it
> into memory in its entirety and then sending it out.
>
> I am using NiFi version 1.12.1
>
> Here's the error from the NiFi logs:
>
>> 2020-10-28 19:34:10,717 ERROR [Timer-Driven Process Thread-6]
>> o.a.n.p.a.storage.PutAzureBlobStorage
>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5]
>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] failed to
>> process session due to java.lang.OutOfMemoryError: Java heap space;
>> Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError:
>> Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 2020-10-28 19:34:10,717 WARN [Timer-Driven Process Thread-6]
>> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
>> PutAzureBlobStorage[id=74b80a47-016d-3430-fd74-ece7653158d5] due to
>> uncaught Exception: java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>>
>
> What's the recommendation for using NiFi to upload files of this size to
> blob storage?
>
> Thanks,
> Eric
>