Ambari server stuck with high CPU util and lots of CLOSE_WAIT connections

[Jeegn Chen <je...@gmail.com>]

Hi all,

Here is how things happened.

I installed Ambari 2.4.2 and deployed a cluster.
Then added some new columns into 2.4.2's database schema and built a
mocked 2.4.5 version.
Made script  change but no database schema change to create a mocked 2.4.7
I used the mocked 2.4.5 and 2.4.7 to exercise ambari-server upgrade.
Upgrading from 2.4.2 to mocked 2.4.5 was fine.
But after I upgraded from my mocked 2.4.5 to my 2.4.7, problems came
out. ambari-server was able to start. But the CPU utilization was
quite high and ambari-server was not responsive. When I checked
'netstat -na | grep CLOSE', I saw a lot of CLOSE_WAIT connections were
related to 127.0.0.1:8080. Nothing unusual was noticed in log.
However, when I installed mock 2.4.7 directly without upgrading,
everything was OK.
Have you happened to noticed similar issue before?
Any hint on how to toubleshoot this problem is appreciated!

Thanks,
Jeegn