You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by srini <na...@gmail.com> on 2017/03/16 18:40:28 UTC
All Partitions have been blacklisted due to failures when
attempting to update. If the Write-Ahead Log is able to perform a
checkpoint, this issue may resolve itself. Otherwise, manual intervention
will be required.
Hi,
We have single nifi instance, and this is our production environment.
Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and
Xmx, and restarted nifi.
Then I see this error [1]. Then I deleted the folder
../nifi-1.1.0/flowfile_repository and restarted the nifi.
Then everything is fine, and it has been working close 20 hours without any
issue.
Now all of sudden, I am seeing the same error. What should I do?
[1] All Partitions have been blacklisted due to failures when attempting to
update. If the Write-Ahead Log is able to perform a checkpoint, this issue
may resolve itself. Otherwise, manual intervention will be required.
thanks
Srini error.txt
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n15167/error.txt>
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Write-Ahead-Lo-tp15167.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual intervention will be required.
Posted by Andy LoPresto <al...@apache.org>.
Srini,
Rather than storing the large amounts of data in the attribute of the original flowfile, can you put them into the content of a new flowfile (correlated with an identifier attribute) and then route the original flowfile on and reference the associated flowfile content when necessary?
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 21, 2017, at 8:41 AM, srini <na...@gmail.com> wrote:
>
> Hello everybody,
> The size of the attribute nifi accepts is a show stopper for me. As my flow
> is big, the flowfile is changes few times. I can't add the attribute (size
> is close to 120KB) to each flowfile. If I put the attribute in the flowfile,
> I can't maintain the same flowfile till the end. As some processes replaces
> the flowfile automatically.
>
> This is a show stopper for me. I have to move all my flow logic to a custom
> processor. That is a huge task.
>
> thanks
> Srini
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Write-Ahead-Lo-tp15167p15207.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: All Partitions have been blacklisted due to failures when
attempting to update. If the Write-Ahead Log is able to perform a
checkpoint, this issue may resolve itself. Otherwise, manual intervention
will be required.
Posted by srini <na...@gmail.com>.
Hello everybody,
The size of the attribute nifi accepts is a show stopper for me. As my flow
is big, the flowfile is changes few times. I can't add the attribute (size
is close to 120KB) to each flowfile. If I put the attribute in the flowfile,
I can't maintain the same flowfile till the end. As some processes replaces
the flowfile automatically.
This is a show stopper for me. I have to move all my flow logic to a custom
processor. That is a huge task.
thanks
Srini
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Write-Ahead-Lo-tp15167p15207.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: All Partitions have been blacklisted due to failures when
attempting to update. If the Write-Ahead Log is able to perform a checkpoint,
this issue may resolve itself. Otherwise, manual intervention will be required.
Posted by James Wing <jv...@gmail.com>.
Would it be possible to configure EvaluateJsonPath to place the selected
JSON fragment in the flowfile content instead of an attribute? Or to break
the selection across multiple attributes rather than one big one?
Thanks,
James
On Thu, Mar 16, 2017 at 11:57 AM, srini <na...@gmail.com> wrote:
> Hi James,
>
> Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I
> do to avoid this?
> thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167p15169.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: All Partitions have been blacklisted due to failures when
attempting to update. If the Write-Ahead Log is able to perform a
checkpoint, this issue may resolve itself. Otherwise, manual intervention
will be required.
Posted by srini <na...@gmail.com>.
Hi James,
Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I
do to avoid this?
thanks
Srini
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Write-Ahead-Lo-tp15167p15169.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: All Partitions have been blacklisted due to failures when
attempting to update. If the Write-Ahead Log is able to perform a checkpoint,
this issue may resolve itself. Otherwise, manual intervention will be required.
Posted by James Wing <jv...@gmail.com>.
Srini,
The error message "FlowFile Repository failed to update" matches a known
issue where NiFi has trouble persisting attributes larger than 64 KB (
https://issues.apache.org/jira/browse/NIFI-3389). Is it possible that your
EvaluateJsonPath is creating attributes exceeding 64 KB? From your error
file:
o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=b71b17e5-1046-115a-5d5c-e4f368141f13] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException:
FlowFile Repository failed to update:
org.apache.nifi.processor.exception.ProcessException: FlowFile Repository
failed to update
Thanks,
James
On Thu, Mar 16, 2017 at 11:40 AM, srini <na...@gmail.com> wrote:
> Hi,
> We have single nifi instance, and this is our production environment.
> Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and
> Xmx, and restarted nifi.
> Then I see this error [1]. Then I deleted the folder
> ../nifi-1.1.0/flowfile_repository and restarted the nifi.
> Then everything is fine, and it has been working close 20 hours without any
> issue.
>
> Now all of sudden, I am seeing the same error. What should I do?
>
> [1] All Partitions have been blacklisted due to failures when attempting to
> update. If the Write-Ahead Log is able to perform a checkpoint, this issue
> may resolve itself. Otherwise, manual intervention will be required.
>
> thanks
> Srini error.txt
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/
> n15167/error.txt>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>