You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Juan Pablo Gardella <ga...@gmail.com> on 2021/03/18 03:44:11 UTC
Any known issue on SplitRecord?
Hi all,
I am using latest nifi version and SplitRecord works only once and then
hangs:
[image: image.png]
I cannot stop it also.
Juan
Re: Any known issue on SplitRecord?
Posted by Joe Witt <jo...@gmail.com>.
I should clarify that I am not positive it is the same issue but it is
certainly possible especially if this worked in 1.13.0
Thanks
On Wed, Mar 17, 2021 at 8:46 PM Joe Witt <jo...@gmail.com> wrote:
> Juan
>
> We found a bug in 1.13.1 today as reported here
> https://issues.apache.org/jira/browse/NIFI-8337 and
> https://issues.apache.org/jira/browse/NIFI-8334.
>
> We will have a 1.13.2 out asap to fix this and the regression now has
> tests to prevent it in the future.
>
> Thanks
> Joe
>
> On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
>> Hi all,
>>
>> I am using latest nifi version and SplitRecord works only once and then
>> hangs:
>>
>> [image: image.png]
>>
>> I cannot stop it also.
>>
>> Juan
>>
>
Re: Any known issue on SplitRecord?
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Hi Mark/Joe
Actually if I wait enough it is able to process the flowfile. When I start
nifi, it is able to process immediately, but the second time I have to wait
around 1/2 minute. I see similar behaviour on Nifi 1.12.1, I have to wait
second time. I don't see the entry log about *Unable to write to container*
but I check my disk and it is almost full đ at least in percentage values:
$ docker exec 373913ea4d00 df -h
Filesystem Size Used Avail Use% Mounted on
overlay 461G 424G 14G 98% /
...
Nothing suspicious in logs. So it is a false alarm, the only problem for me
is understanding why it takes so long to split an Avro record (result from
ExecuteSQLRecord).
Juan
On Thu, 18 Mar 2021 at 11:43, Mark Payne <ma...@hotmail.com> wrote:
> Juan,
>
> How full is your disk that holds your content repository?
> The issue that Joe mentioned, I donât think, will cause what youâre
> seeing. But I am guessing that your content repository is exerting back
> pressure.
>
> Can you check your logs for the message "Unable to write to container"? I
> am guessing that you will see a message along the lines of:
>
> 2021-03-18 10:26:20,988 INFO [Timer-Driven Process Thread-10] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup
>
> This would indicate that your content repository is filling up and is
> exerting backpressure on the Input Port, which prevents it from writing the
> data out. This can potentially result in the transfer timing out.
>
> By default, the content repository exerts backpressure if the disk is 50%
> full. This can be configured in nifi.properties:
>
> nifi.content.repository.archive.max.usage.percentage=50%
>
>
> Thanks
> -Mark
>
>
> On Mar 17, 2021, at 11:58 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Thanks Juan - that would be very valuable actually. I'll send you a link
> to a build here in an hour or so. If you can test that and let us know that
> will help us with the release candidate voting process quite a bit.
>
> Thanks
>
> On Wed, Mar 17, 2021 at 8:49 PM Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
>> Wow that is fast! You are awesome, thanks Joe. I will test it.
>>
>> Juan
>>
>> On Thu, 18 Mar 2021 at 00:47, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Juan
>>>
>>> We found a bug in 1.13.1 today as reported here
>>> https://issues.apache.org/jira/browse/NIFI-8337 and
>>> https://issues.apache.org/jira/browse/NIFI-8334.
>>>
>>> We will have a 1.13.2 out asap to fix this and the regression now has
>>> tests to prevent it in the future.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <
>>> gardellajuanpablo@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am using latest nifi version and SplitRecord works only once and then
>>>> hangs:
>>>>
>>>> <image.png>
>>>>
>>>> I cannot stop it also.
>>>>
>>>> Juan
>>>>
>>>
>
Re: Any known issue on SplitRecord?
Posted by Mark Payne <ma...@hotmail.com>.
Juan,
How full is your disk that holds your content repository?
The issue that Joe mentioned, I donât think, will cause what youâre seeing. But I am guessing that your content repository is exerting back pressure.
Can you check your logs for the message "Unable to write to container"? I am guessing that you will see a message along the lines of:
2021-03-18 10:26:20,988 INFO [Timer-Driven Process Thread-10] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup
This would indicate that your content repository is filling up and is exerting backpressure on the Input Port, which prevents it from writing the data out. This can potentially result in the transfer timing out.
By default, the content repository exerts backpressure if the disk is 50% full. This can be configured in nifi.properties:
nifi.content.repository.archive.max.usage.percentage=50%
Thanks
-Mark
On Mar 17, 2021, at 11:58 PM, Joe Witt <jo...@gmail.com>> wrote:
Thanks Juan - that would be very valuable actually. I'll send you a link to a build here in an hour or so. If you can test that and let us know that will help us with the release candidate voting process quite a bit.
Thanks
On Wed, Mar 17, 2021 at 8:49 PM Juan Pablo Gardella <ga...@gmail.com>> wrote:
Wow that is fast! You are awesome, thanks Joe. I will test it.
Juan
On Thu, 18 Mar 2021 at 00:47, Joe Witt <jo...@gmail.com>> wrote:
Juan
We found a bug in 1.13.1 today as reported here https://issues.apache.org/jira/browse/NIFI-8337 and https://issues.apache.org/jira/browse/NIFI-8334.
We will have a 1.13.2 out asap to fix this and the regression now has tests to prevent it in the future.
Thanks
Joe
On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hi all,
I am using latest nifi version and SplitRecord works only once and then hangs:
<image.png>
I cannot stop it also.
Juan
Re: Any known issue on SplitRecord?
Posted by Joe Witt <jo...@gmail.com>.
Thanks Juan - that would be very valuable actually. I'll send you a link
to a build here in an hour or so. If you can test that and let us know that
will help us with the release candidate voting process quite a bit.
Thanks
On Wed, Mar 17, 2021 at 8:49 PM Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> Wow that is fast! You are awesome, thanks Joe. I will test it.
>
> Juan
>
> On Thu, 18 Mar 2021 at 00:47, Joe Witt <jo...@gmail.com> wrote:
>
>> Juan
>>
>> We found a bug in 1.13.1 today as reported here
>> https://issues.apache.org/jira/browse/NIFI-8337 and
>> https://issues.apache.org/jira/browse/NIFI-8334.
>>
>> We will have a 1.13.2 out asap to fix this and the regression now has
>> tests to prevent it in the future.
>>
>> Thanks
>> Joe
>>
>> On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <
>> gardellajuanpablo@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am using latest nifi version and SplitRecord works only once and then
>>> hangs:
>>>
>>> [image: image.png]
>>>
>>> I cannot stop it also.
>>>
>>> Juan
>>>
>>
Re: Any known issue on SplitRecord?
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Wow that is fast! You are awesome, thanks Joe. I will test it.
Juan
On Thu, 18 Mar 2021 at 00:47, Joe Witt <jo...@gmail.com> wrote:
> Juan
>
> We found a bug in 1.13.1 today as reported here
> https://issues.apache.org/jira/browse/NIFI-8337 and
> https://issues.apache.org/jira/browse/NIFI-8334.
>
> We will have a 1.13.2 out asap to fix this and the regression now has
> tests to prevent it in the future.
>
> Thanks
> Joe
>
> On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
>> Hi all,
>>
>> I am using latest nifi version and SplitRecord works only once and then
>> hangs:
>>
>> [image: image.png]
>>
>> I cannot stop it also.
>>
>> Juan
>>
>
Re: Any known issue on SplitRecord?
Posted by Joe Witt <jo...@gmail.com>.
Juan
We found a bug in 1.13.1 today as reported here
https://issues.apache.org/jira/browse/NIFI-8337 and
https://issues.apache.org/jira/browse/NIFI-8334.
We will have a 1.13.2 out asap to fix this and the regression now has tests
to prevent it in the future.
Thanks
Joe
On Wed, Mar 17, 2021 at 8:44 PM Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> Hi all,
>
> I am using latest nifi version and SplitRecord works only once and then
> hangs:
>
> [image: image.png]
>
> I cannot stop it also.
>
> Juan
>