You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Gresock <jg...@gmail.com> on 2017/06/14 15:41:55 UTC
NiFi 1.3.0 Remote ports "disabled" after a while?
Before I investigate further, has anyone encountered NiFi RPGs "disabling"
after a while in 1.3.0? Their status is reported as enabled/transmitting,
but the flow files just sit there until I "enable transmission" again.
This happens for nearly all of the RPGs I have in my flow. I just have to
keep "kick starting" them periodically to keep things flowing.
This behavior did not happen in 1.2.0.
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Posted by Joe Gresock <jg...@gmail.com>.
Great, thanks Bryan.
On Thu, Jun 15, 2017 at 2:38 PM, Bryan Bende <bb...@gmail.com> wrote:
> Joe,
>
> Thanks for the info.
>
> I think this might be similar to
> https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't
> account for RPGs.
>
> I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075
>
> -Bryan
>
> On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <jg...@gmail.com> wrote:
> > Btw, I was wrong about one statement in my last email.. this particular
> RPG
> > (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes
> are
> > STOPPED and some are RUNNING. However, even on the nodes that show
> RUNNING
> > in the flow.xml.gz, the queue is still blocked.
> >
> >
> >
> > On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <jg...@gmail.com> wrote:
> >
> >> Great questions, Koji:
> >>
> >> 1. RAW
> >> 2. Push
> >> 3. No, compression is not enabled
> >> 4. Several, but it has the same effect to "Enable transmission" as it
> does
> >> to specifically disable and then enable a specific port. The key being
> >> that all the ports are marked as enabled to start, even when not
> sending.
> >> 5. This is anecdotal, but it feels like 1+ times per day.
> >> 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There
> are
> >> multiple self-RPGs, including this one, at various points on the graph.
> >> 7. Attached. The relevant queue that is inexplicably blocked shows this
> >> in the log:
> >> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
> >> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
> >> TIMED_WAITING on java.util.concurrent.locks.
> AbstractQueuedSynchronizer$
> >> ConditionObject@2edb0217
> >> at sun.misc.Unsafe.park(Native Method)
> >> at java.util.concurrent.locks.LockSupport.parkNanos(
> >> LockSupport.java:215)
> >> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >> at java.util.concurrent.ScheduledThreadPoolExecutor$
> >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >> at java.util.concurrent.ScheduledThreadPoolExecutor$
> >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >> at java.util.concurrent.ThreadPoolExecutor.getTask(
> >> ThreadPoolExecutor.java:1067)
> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> >> ThreadPoolExecutor.java:1127)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >> ThreadPoolExecutor.java:617)
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> 8. No proxy
> >>
> >> One strange thing I noticed was that the <scheduledState> is not
> >> consistent in the flow.xml.gz between some of the nodes for other
> instances
> >> of this self-RPG (but not for the one that is currently backing up):
> >> 1. On disk, some of the nodes have this input port set to STOPPED, and
> >> some have it set to RUNNING.
> >> 2. In the UI, the input port is displayed as RUNNING.
> >>
> >>
> >>
> >> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ijokarumawak@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Joe,
> >>>
> >>> I haven't encountered that issue. Would you tell us more details? So
> >>> that we can investigate it.
> >>>
> >>> 1. Which transport protocol are you using? RAW or HTTP
> >>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
> >>> remote OutputPort, Push = sends to remote InputPort
> >>> 3. Is compression enabled?
> >>> 4. How many ports are configured? If there are more than one, RPG
> >>> indicates that it's 'Transmitting' as long as at least one port is
> >>> transmitting, while others can be disabled. Right click a RPG then
> >>> select 'Remote Ports' to see each port status.
> >>> 5. How often does it happen? Stops once per hour/day/week??
> >>> 6. Please share your environment, OS, Java version ... etc
> >>> 7. If possible, please take stack-trace while you are facing that
> >>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
> >>> stack-trace will be written to nifi-bootstrap.log
> >>> 8. Is proxy used?
> >>>
> >>> Thanks,
> >>> Koji
> >>>
> >>>
> >>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jg...@gmail.com>
> wrote:
> >>> > Before I investigate further, has anyone encountered NiFi RPGs
> >>> "disabling"
> >>> > after a while in 1.3.0? Their status is reported as
> >>> enabled/transmitting,
> >>> > but the flow files just sit there until I "enable transmission"
> again.
> >>> > This happens for nearly all of the RPGs I have in my flow. I just
> have
> >>> to
> >>> > keep "kick starting" them periodically to keep things flowing.
> >>> >
> >>> > This behavior did not happen in 1.2.0.
> >>> > --
> >>> > I know what it is to be in need, and I know what it is to have
> plenty.
> >>> I
> >>> > have learned the secret of being content in any and every situation,
> >>> > whether well fed or hungry, whether living in plenty or in want. I
> can
> >>> do
> >>> > all this through him who gives me strength. *-Philippians 4:12-13*
> >>>
> >>
> >>
> >>
> >> --
> >> I know what it is to be in need, and I know what it is to have plenty.
> I
> >> have learned the secret of being content in any and every situation,
> >> whether well fed or hungry, whether living in plenty or in want. I can
> >> do all this through him who gives me strength. *-Philippians 4:12-13*
> >>
> >
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty. I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want. I can
> do
> > all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Posted by Bryan Bende <bb...@gmail.com>.
Joe,
Thanks for the info.
I think this might be similar to
https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't
account for RPGs.
I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075
-Bryan
On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <jg...@gmail.com> wrote:
> Btw, I was wrong about one statement in my last email.. this particular RPG
> (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are
> STOPPED and some are RUNNING. However, even on the nodes that show RUNNING
> in the flow.xml.gz, the queue is still blocked.
>
>
>
> On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <jg...@gmail.com> wrote:
>
>> Great questions, Koji:
>>
>> 1. RAW
>> 2. Push
>> 3. No, compression is not enabled
>> 4. Several, but it has the same effect to "Enable transmission" as it does
>> to specifically disable and then enable a specific port. The key being
>> that all the ports are marked as enabled to start, even when not sending.
>> 5. This is anecdotal, but it feels like 1+ times per day.
>> 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There are
>> multiple self-RPGs, including this one, at various points on the graph.
>> 7. Attached. The relevant queue that is inexplicably blocked shows this
>> in the log:
>> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
>> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
>> TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject@2edb0217
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:215)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1067)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> 8. No proxy
>>
>> One strange thing I noticed was that the <scheduledState> is not
>> consistent in the flow.xml.gz between some of the nodes for other instances
>> of this self-RPG (but not for the one that is currently backing up):
>> 1. On disk, some of the nodes have this input port set to STOPPED, and
>> some have it set to RUNNING.
>> 2. In the UI, the input port is displayed as RUNNING.
>>
>>
>>
>> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ij...@gmail.com>
>> wrote:
>>
>>> Hi Joe,
>>>
>>> I haven't encountered that issue. Would you tell us more details? So
>>> that we can investigate it.
>>>
>>> 1. Which transport protocol are you using? RAW or HTTP
>>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
>>> remote OutputPort, Push = sends to remote InputPort
>>> 3. Is compression enabled?
>>> 4. How many ports are configured? If there are more than one, RPG
>>> indicates that it's 'Transmitting' as long as at least one port is
>>> transmitting, while others can be disabled. Right click a RPG then
>>> select 'Remote Ports' to see each port status.
>>> 5. How often does it happen? Stops once per hour/day/week??
>>> 6. Please share your environment, OS, Java version ... etc
>>> 7. If possible, please take stack-trace while you are facing that
>>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
>>> stack-trace will be written to nifi-bootstrap.log
>>> 8. Is proxy used?
>>>
>>> Thanks,
>>> Koji
>>>
>>>
>>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jg...@gmail.com> wrote:
>>> > Before I investigate further, has anyone encountered NiFi RPGs
>>> "disabling"
>>> > after a while in 1.3.0? Their status is reported as
>>> enabled/transmitting,
>>> > but the flow files just sit there until I "enable transmission" again.
>>> > This happens for nearly all of the RPGs I have in my flow. I just have
>>> to
>>> > keep "kick starting" them periodically to keep things flowing.
>>> >
>>> > This behavior did not happen in 1.2.0.
>>> > --
>>> > I know what it is to be in need, and I know what it is to have plenty.
>>> I
>>> > have learned the secret of being content in any and every situation,
>>> > whether well fed or hungry, whether living in plenty or in want. I can
>>> do
>>> > all this through him who gives me strength. *-Philippians 4:12-13*
>>>
>>
>>
>>
>> --
>> I know what it is to be in need, and I know what it is to have plenty. I
>> have learned the secret of being content in any and every situation,
>> whether well fed or hungry, whether living in plenty or in want. I can
>> do all this through him who gives me strength. *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*
Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Posted by Joe Gresock <jg...@gmail.com>.
Btw, I was wrong about one statement in my last email.. this particular RPG
(705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are
STOPPED and some are RUNNING. However, even on the nodes that show RUNNING
in the flow.xml.gz, the queue is still blocked.
On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <jg...@gmail.com> wrote:
> Great questions, Koji:
>
> 1. RAW
> 2. Push
> 3. No, compression is not enabled
> 4. Several, but it has the same effect to "Enable transmission" as it does
> to specifically disable and then enable a specific port. The key being
> that all the ports are marked as enabled to start, even when not sending.
> 5. This is anecdotal, but it feels like 1+ times per day.
> 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There are
> multiple self-RPGs, including this one, at various points on the graph.
> 7. Attached. The relevant queue that is inexplicably blocked shows this
> in the log:
> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
> TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject@2edb0217
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
> 8. No proxy
>
> One strange thing I noticed was that the <scheduledState> is not
> consistent in the flow.xml.gz between some of the nodes for other instances
> of this self-RPG (but not for the one that is currently backing up):
> 1. On disk, some of the nodes have this input port set to STOPPED, and
> some have it set to RUNNING.
> 2. In the UI, the input port is displayed as RUNNING.
>
>
>
> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ij...@gmail.com>
> wrote:
>
>> Hi Joe,
>>
>> I haven't encountered that issue. Would you tell us more details? So
>> that we can investigate it.
>>
>> 1. Which transport protocol are you using? RAW or HTTP
>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
>> remote OutputPort, Push = sends to remote InputPort
>> 3. Is compression enabled?
>> 4. How many ports are configured? If there are more than one, RPG
>> indicates that it's 'Transmitting' as long as at least one port is
>> transmitting, while others can be disabled. Right click a RPG then
>> select 'Remote Ports' to see each port status.
>> 5. How often does it happen? Stops once per hour/day/week??
>> 6. Please share your environment, OS, Java version ... etc
>> 7. If possible, please take stack-trace while you are facing that
>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
>> stack-trace will be written to nifi-bootstrap.log
>> 8. Is proxy used?
>>
>> Thanks,
>> Koji
>>
>>
>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jg...@gmail.com> wrote:
>> > Before I investigate further, has anyone encountered NiFi RPGs
>> "disabling"
>> > after a while in 1.3.0? Their status is reported as
>> enabled/transmitting,
>> > but the flow files just sit there until I "enable transmission" again.
>> > This happens for nearly all of the RPGs I have in my flow. I just have
>> to
>> > keep "kick starting" them periodically to keep things flowing.
>> >
>> > This behavior did not happen in 1.2.0.
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.
>> I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want. I can
>> do
>> > all this through him who gives me strength. *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can
> do all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Posted by Joe Gresock <jg...@gmail.com>.
Great questions, Koji:
1. RAW
2. Push
3. No, compression is not enabled
4. Several, but it has the same effect to "Enable transmission" as it does
to specifically disable and then enable a specific port. The key being
that all the ports are marked as enabled to start, even when not sending.
5. This is anecdotal, but it feels like 1+ times per day.
6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There are
multiple self-RPGs, including this one, at various points on the graph.
7. Attached. The relevant queue that is inexplicably blocked shows this in
the log:
"Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edb0217
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
8. No proxy
One strange thing I noticed was that the <scheduledState> is not consistent
in the flow.xml.gz between some of the nodes for other instances of this
self-RPG (but not for the one that is currently backing up):
1. On disk, some of the nodes have this input port set to STOPPED, and some
have it set to RUNNING.
2. In the UI, the input port is displayed as RUNNING.
On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ij...@gmail.com>
wrote:
> Hi Joe,
>
> I haven't encountered that issue. Would you tell us more details? So
> that we can investigate it.
>
> 1. Which transport protocol are you using? RAW or HTTP
> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
> remote OutputPort, Push = sends to remote InputPort
> 3. Is compression enabled?
> 4. How many ports are configured? If there are more than one, RPG
> indicates that it's 'Transmitting' as long as at least one port is
> transmitting, while others can be disabled. Right click a RPG then
> select 'Remote Ports' to see each port status.
> 5. How often does it happen? Stops once per hour/day/week??
> 6. Please share your environment, OS, Java version ... etc
> 7. If possible, please take stack-trace while you are facing that
> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
> stack-trace will be written to nifi-bootstrap.log
> 8. Is proxy used?
>
> Thanks,
> Koji
>
>
> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jg...@gmail.com> wrote:
> > Before I investigate further, has anyone encountered NiFi RPGs
> "disabling"
> > after a while in 1.3.0? Their status is reported as
> enabled/transmitting,
> > but the flow files just sit there until I "enable transmission" again.
> > This happens for nearly all of the RPGs I have in my flow. I just have
> to
> > keep "kick starting" them periodically to keep things flowing.
> >
> > This behavior did not happen in 1.2.0.
> > --
> > I know what it is to be in need, and I know what it is to have plenty. I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want. I can
> do
> > all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Joe,
I haven't encountered that issue. Would you tell us more details? So
that we can investigate it.
1. Which transport protocol are you using? RAW or HTTP
2. Is this a pull or push transfer? Pull = client NiFi pulls from
remote OutputPort, Push = sends to remote InputPort
3. Is compression enabled?
4. How many ports are configured? If there are more than one, RPG
indicates that it's 'Transmitting' as long as at least one port is
transmitting, while others can be disabled. Right click a RPG then
select 'Remote Ports' to see each port status.
5. How often does it happen? Stops once per hour/day/week??
6. Please share your environment, OS, Java version ... etc
7. If possible, please take stack-trace while you are facing that
issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
stack-trace will be written to nifi-bootstrap.log
8. Is proxy used?
Thanks,
Koji
On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jg...@gmail.com> wrote:
> Before I investigate further, has anyone encountered NiFi RPGs "disabling"
> after a while in 1.3.0? Their status is reported as enabled/transmitting,
> but the flow files just sit there until I "enable transmission" again.
> This happens for nearly all of the RPGs I have in my flow. I just have to
> keep "kick starting" them periodically to keep things flowing.
>
> This behavior did not happen in 1.2.0.
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*