You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Dmitry Ibragimov (Jira)" <ji...@apache.org> on 2020/05/09 20:11:00 UTC
[jira] [Created] (NIFI-7437) UI is slow when
nifi.analytics.predict.enabled is true
Dmitry Ibragimov created NIFI-7437:
--------------------------------------
Summary: UI is slow when nifi.analytics.predict.enabled is true
Key: NIFI-7437
URL: https://issues.apache.org/jira/browse/NIFI-7437
Project: Apache NiFi
Issue Type: Bug
Components: Core UI, Extensions
Affects Versions: 1.11.4, 1.10.0
Environment: Java11, CentOS8
Reporter: Dmitry Ibragimov
We faced with issue when nifi.analytics.predict.enabled is true after cluster upgrade to 1.11.4
We have about 4000 processors in development enviroment, but most of them is in disabled state: 256 running, 1263 stopped, 2543 disabled
After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure prediction feature and enable it in configuration:
{code:java}
nifi.analytics.predict.enabled=true
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90
{code}
And we faced with terrible UI performance degradataion. Root page opens in 20 seconds instead of 200-500ms. About ~100 times slower. I've tesed it with different environments centos7/8, java8/11, clustered secured, clustered unsecured, standalone unsecured - all the same.
In debug log for ThreadPoolRequestReplicator:
{code:java}
2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2] o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET /nifi-api/flow/process-groups/root (Request ID c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max = 20625, average = 20161.0 ms
2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET /nifi-api/flow/process-groups/root (Request ID c144196f-d4cb-4053-8828-70e06f7c5100):
newnifi01:8080: 19548 millis
newnifi02:8080: 20625 millis
newnifi03:8080: 20310 millis{code}
More deep debug:
{code:java}
2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21] org.eclipse.jetty.server.HttpChannel REQUEST for //newnifi01:8080/nifi-api/flow/process-groups/root on HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
Host: newnifi01:8080
...
2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time back pressure by content size in bytes. Returning -1
2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time to back pressure by object count. Returning -1
2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting content size in bytes for next interval. Returning -1
2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting object count for next interval. Returning -1
2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting object count for next interval. Returning -1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting content size in bytes for next interval. Returning -1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: nextIntervalPercentageUseCount=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: nextIntervalBytes=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: timeToBytesBackpressureMillis=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: nextIntervalCount=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: nextIntervalPercentageUseBytes=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: intervalTimeMillis=180000
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id eb602b2a-016f-1000-0000-00002767192a: timeToCountBackpressureMillis=-1
2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.CachingConnectionStatusAnalyticsEngine Pulled existing analytics from cache for connection id: ec014ca8-a82b-10bb-0000-00004069f95e
2020-05-09 10:31:13,260 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time back pressure by content size in bytes. Returning -1
2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time to back pressure by object count. Returning -1
2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting content size in bytes for next interval. Returning -1
2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting object count for next interval. Returning -1
2020-05-09 10:31:13,262 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting object count for next interval. Returning -1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting content size in bytes for next interval. Returning -1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalPercentageUseCount=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalBytes=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: timeToBytesBackpressureMillis=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalCount=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalPercentageUseBytes=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: intervalTimeMillis=180000
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id ec014ca8-a82b-10bb-0000-00004069f95e: timeToCountBackpressureMillis=-1
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.CachingConnectionStatusAnalyticsEngine Pulled existing analytics from cache for connection id: eb61f002-016f-1000-0000-000067a860ea
2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time back pressure by content size in bytes. Returning -1
...
A lot of same messages
...
2020-05-09 10:31:32,758 DEBUG [NiFi Web Server-21] org.eclipse.jetty.server.HttpChannel COMMIT for /nifi-api/flow/process-groups/root on HttpChannelOverHttp@68d3e945{r=1,c=true,c=false/false,a=DISPATCHED,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=19506}
200 OK HTTP/1.1
Date: Sat, 09 May 2020 07:31:13 GMT
{code}
3ms prediction is a quite fast, but if calculate count this messages it is the same as number of processors:
{code:java}
#zgrep "2020-05-09 10:31:" /var/log/nifi/nifi-app_2020-05-09_10.1.log.gz | grep "Prediction model for connection id" | cut -d' ' -f 13 | sort | uniq -c | wc -l
4842{code}
I've checked several random connection id's - it is from and to disabled processors.
Maybe need to skip prediction back-pressure for disbled connections? Any suppouse, how can we fix it without disable prediction entirely?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)