You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Juha (Jira)" <ji...@apache.org> on 2022/08/12 09:43:00 UTC
[jira] [Created] (FLINK-28947) Curator framework fails with NullPointerException
Juha created FLINK-28947:
----------------------------
Summary: Curator framework fails with NullPointerException
Key: FLINK-28947
URL: https://issues.apache.org/jira/browse/FLINK-28947
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.1
Reporter: Juha
I'm getting the following error in JobManager and as a result JobManager exits.
{code:java}
Aug 12 06:37:30 server_name java[173]: [2022-08-12 06:37:30,491] ERROR Background exception was not retry-able or retry gave up (org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl:733)
Aug 12 06:37:30 server_name java[173]: java.lang.NullPointerException: null
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress(Compatibility.java:116) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString(EnsembleTracker.java:185) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:206) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300(EnsembleTracker.java:50) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:150) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:926) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:683) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: [2022-08-12 06:37:30,493] ERROR Unhandled error in curator framework, error message: Background exception was not retry-able or retry gave up (org.apache.flink.runtime.util.ZooKeeperUtils:292)
Aug 12 06:37:30 server_name java[173]: java.lang.NullPointerException: null
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress(Compatibility.java:116) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString(EnsembleTracker.java:185) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:206) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300(EnsembleTracker.java:50) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:150) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:926) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:683) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: [2022-08-12 06:37:30,494] ERROR Fatal error occurred while executing the TaskManager. Shutting it down... (org.apache.flink.runtime.taskexecutor.TaskManagerRunner:427)
Aug 12 06:37:30 server_name java[173]: java.lang.NullPointerException: null
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress(Compatibility.java:116) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString(EnsembleTracker.java:185) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:206) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300(EnsembleTracker.java:50) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:150) ~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:926) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:683) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
{code}
Steps
* Create three servers
* Run Flink JobManager and TaskManager on all of them (let's call these A, B and C). Use ZooKeeper HA Services.
* Everything works as expected
* Add a new server (D).
* Shutdown server C
* This error can be seen on both servers A and D. I didn't check B and C.
This can be reproduced (apparently) with every execution.
I'm using Flink 1.15.1. Actually I'm migrating from 1.13.X to 1.15.X. I'm not totally sure whether this ever happens on 1.13.X, but it seems to _always_ happen on 1.15.1.
I looked using debugger what's going on in the JobManager:
{code:java}
main-EventThread[1] where
[1] org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress (Compatibility.java:116)
[2] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString (EnsembleTracker.java:185)
[3] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData (EnsembleTracker.java:206)
[4] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300 (EnsembleTracker.java:50)
[5] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult (EnsembleTracker.java:150)
[6] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback (CuratorFrameworkImpl.java:926)
[7] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation (CuratorFrameworkImpl.java:683)
[8] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation (WatcherRemovalFacade.java:152)
[9] org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult (GetConfigBuilderImpl.java:222)
[10] org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.processEvent (ClientCnxn.java:598)
[11] org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:510)
main-EventThread[1] dump address
address = {
holder: instance of java.net.InetSocketAddress$InetSocketAddressHolder(id=8302)
serialVersionUID: 5076001401234631237
serialPersistentFields: instance of java.io.ObjectStreamField[3] (id=8303)
UNSAFE: instance of jdk.internal.misc.Unsafe(id=8304)
FIELDS_OFFSET: 12
java.net.SocketAddress.serialVersionUID: 5215720748342549866
}
main-EventThread[1] dump address.holder
address.holder = {
hostname: "host_name_here"
addr: null
port: 2888
}
main-EventThread[1] print address.getAddress()
address.getAddress() = null
{code}
(The hostname has been changed).
It can be seen that on line 116 of Compatibility.java (https://github.com/apache/curator/blob/d65669b64f003326c98843b32b997e3ffab1e442/curator-client/src/main/java/org/apache/curator/utils/Compatibility.java#L116) there's this
{code}
return (address != null) ? address.getAddress().getHostAddress() : "unknown";
{code}
Here {{address.getAddress()}} returns {{null}} causing the crash.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)