You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "David Handermann (Jira)" <ji...@apache.org> on 2025/04/10 14:27:00 UTC
[jira] [Commented] (NIFI-14462) Regular getting HTTP 502 Error behind Apache Reverse Proxy
[ https://issues.apache.org/jira/browse/NIFI-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17943237#comment-17943237 ]
David Handermann commented on NIFI-14462:
-----------------------------------------
Thanks for the thorough description and workaround details.
As evidenced by the disabling of connection reuse, the core issue sounds like NiFi closing the connection after a period of time. NiFi 1 used Jetty 9 and NiFi 2 uses Jetty 12, so there have been quite a few changes between these versions.
Reviewing the [ProxyPass documentation|https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass] one other parameter worth trying is the {{ttl}} property. The idle timeout in NiFi Jetty Server is 60 seconds, so the {{ttl}} property should be set to 60 seconds or less. That may allow connection pooling to be used without the other workarounds.
> Regular getting HTTP 502 Error behind Apache Reverse Proxy
> ----------------------------------------------------------
>
> Key: NIFI-14462
> URL: https://issues.apache.org/jira/browse/NIFI-14462
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 2.3.0
> Environment: Debian 11/12
> Podman 5.3.0/5.4.2
> Apache2: 2.4.62
> Reporter: Fabian Reiber
> Priority: Major
>
> We are using nifi in a secured cluster witch three nodes behind an apache reverse proxy. After the update from 1.28.1 to 2.3.0, we regularly get 502 proxy errors like this in the browser:
> {code:none}
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> <hr> <address>Apache/2.4.62 (Debian) Server at nifi.local.example.com Port 443</address> </body></html> {code}
> This did not happen with nifi 1. Nifi fetches various api endpoints while running. And each of the endpoint can throw this error. Most of the time the requests are ok and nifi runs as expected. But sometimes not. Here is an example of the apache error log, after we raise the log level to trace5:
> {code:none}
> [Thu Apr 10 14:58:01.694121 2025] [proxy:trace2] [pid 14770:tid 14793] mod_proxy.c(848): [client 192.168.58.1:38278] AH03461: attempting to match URI path '/nifi-api/flow/status' against prefix '/' for proxying, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694125 2025] [proxy:debug] [pid 14770:tid 14784] mod_proxy.c(1465): [client 192.168.58.1:38262] AH01143: Running scheme https handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694139 2025] [proxy:trace1] [pid 14770:tid 14793] mod_proxy.c(969): [client 192.168.58.1:38278] AH03464: URI path '/nifi-api/flow/status' matches proxy handler 'proxy:https://nifi.local.example.com:8443/nifi-api/flow/status', referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694143 2025] [proxy:debug] [pid 14770:tid 14784] proxy_util.c(2797): AH00942: https: has acquired connection for (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694149 2025] [proxy:debug] [pid 14770:tid 14784] proxy_util.c(3242): [client 192.168.58.1:38262] AH00944: connecting https://nifi.local.example.com:8443/nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true to nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694161 2025] [proxy:debug] [pid 14770:tid 14784] proxy_util.c(3450): [client 192.168.58.1:38262] AH00947: connecting /nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true to 127.0.0.1:8443 (nifi.local.example.com:8443), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694187 2025] [proxy:trace2] [pid 14770:tid 14784] proxy_util.c(3734): https: reusing backend connection 127.0.0.1:42282<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694279 2025] [proxy:trace2] [pid 14770:tid 14793] proxy_util.c(2604): [client 192.168.58.1:38278] https: found worker https://nifi.local.example.com:8443/ for https://nifi.local.example.com:8443/nifi-api/flow/status, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694300 2025] [proxy:debug] [pid 14770:tid 14793] mod_proxy.c(1465): [client 192.168.58.1:38278] AH01143: Running scheme https handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694328 2025] [proxy:debug] [pid 14770:tid 14793] proxy_util.c(2797): AH00942: https: has acquired connection for (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694334 2025] [proxy:debug] [pid 14770:tid 14793] proxy_util.c(3242): [client 192.168.58.1:38278] AH00944: connecting https://nifi.local.example.com:8443/nifi-api/flow/status to nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694340 2025] [proxy:debug] [pid 14770:tid 14793] proxy_util.c(3450): [client 192.168.58.1:38278] AH00947: connecting /nifi-api/flow/status to 127.0.0.1:8443 (nifi.local.example.com:8443), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694745 2025] [proxy:trace2] [pid 14770:tid 14793] proxy_util.c(3734): https: reusing backend connection 127.0.0.1:42272<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694887 2025] [proxy_http:error] [pid 14770:tid 14793] (70014)End of file found: [client 192.168.58.1:38278] AH01102: error reading status line from remote server nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694905 2025] [proxy:error] [pid 14770:tid 14793] [client 192.168.58.1:38278] AH00898: Error reading from remote server returned by /nifi-api/flow/status, referer: https://nifi.local.example.com/nifi/{code}
> Actually the last two lines are the most interesting, but don't give much information.
> We can reproduce this behavior on two different clusters on different virtual machines with different debian and podman versions as well as with a secured single node instance from scratch. We are using podman to run nifi in a container/pod. We have our own certificate authority to configure the proxy with certificates. But I think this is not important in this case. So, the location directive of the apache config of the single node instance, which is more interesting I guess, looks like this:
> {code:none}
> <Location "/">
> ProxyPass https://nifi.local.example.com:8443/
> ProxyPassReverse https://nifi.local.example.com:8443/
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "nifi.local.example.com"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/"
> </Location>{code}
> This part does not differ a lot to the config of the webserver in our cluster.
> But at least we found two workarounds. If we set "SetEnv proxy-initial-not-pooled 1" in the location directive of the apache config, we do not get any proxy error again and nifi works as expected. Another second possibility is by setting "disablereuse=On" in the "ProxyPass" directive.
> The documentation of mod_proxy says about that variable:
> ??This avoids the "proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend.??
> So, if no pooled connection is reused or not any connection at all is reused (with "disablereuse=On"), nifi does not close the connection to early to the proxy.
> For more information take a look at the apache documentation for [mod_proxy Environmental variables|https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html#env] and [ProxyPass|https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#ProxyPass]
> If you need more information please let me know :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)