You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Mario Salazar de Torres (Jira)" <ji...@apache.org> on 2022/02/04 08:47:00 UTC
[jira] [Comment Edited] (GEODE-10017) Fix new ITs unstability for TCs that involve members restart
[ https://issues.apache.org/jira/browse/GEODE-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486856#comment-17486856 ]
Mario Salazar de Torres edited comment on GEODE-10017 at 2/4/22, 8:46 AM:
--------------------------------------------------------------------------
After some digging I noticed that for the executions on which one of these TCs get stuck, GFSH tool running the "start server ..." command, never returns.
So I checked why GFSH was not returning and I saw that it was stuck checking the server state, which was always returning as non-responding, even whenever there was a server instance running (as I could see by connecting to the cluster and running list member)/listing processes in the test container/looking at the logs.
So, looking for all of the possible scenarios which could cause ServerState to show up as "Not responding" I noticed that there were no PID file inside the recently started server and this was causing the whole issue.
Now, my theory for why that's happening is the following:
# First instance of the server is notified to be stopped.
# Cache for the first server is closed, and GFSH process stopping the server exists, returning the control to the test process.
# Given that the from the point of view of the test the server has been stopped already, it runs the startup for the new server.
# The newer server instance writes the PID file with its PID.
# The older server instance, which was still running, deletes its PID along some other files and actually terminate its process.
The time gap between step 4 and 5 normally is really tight, so that would explain why this issue only reproduces sometimes and mostly on overloaded systems.
was (Author: gaussianrecurrence):
After some digging I noticed that for the executions on which one of these TCs get stuck, GFSH tool running the "start server ..." command, never returns.
So I checked why GFSH was not returning and I saw that it was stuck checking the server state, which was always returning as non-responding, even whenever there was a server instance running (as I could see by connecting to the cluster and running list member)/listing processes in the test container/looking at the logs.
So, looking for all of the possible scenarios which could cause ServerState to show up as "Not responding" I noticed that there were no PID file inside the recently started server and this was causing the whole issue.
Now, my theory for why that's happening is the following:
# First instance of the server is notified to be stopped.
# Cache for the first server is closed, and GFSH process stopping the server exists, returning the control to the test process.
# Given that the from the point of view of the test the server has been stopped already, it runs the startup for the new server.
# The newer server instance writes the PID file with its PID.
# The older server instance, which was still running, deletes its PID along some other files and actually terminate its process.
The time gap between step 4 and 5 normally is really tight, so that would explain why this issue only reproduces sometimes and mostly on overloaded systems.
> Fix new ITs unstability for TCs that involve members restart
> ------------------------------------------------------------
>
> Key: GEODE-10017
> URL: https://issues.apache.org/jira/browse/GEODE-10017
> Project: Geode
> Issue Type: Bug
> Components: native client
> Reporter: Mario Salazar de Torres
> Priority: Major
> Labels: needsTriage
>
> *GIVEN* an integration TC on which a server restart needs to be restarted
> *WHEN* the server is starting up again
> *THEN* +{color:#172b4d}it might{color}+{color:#172b4d} happen that the TC gets stuck{color}
> ----
> *Additional information.* This issue does not always happens, and I've seen it happening more frequently with the latest version of Geode server (1.15.0)
> Some examples of this TC are:
> * RegisterKeysTest.RegisterKeySetAndClusterRestart
> * PartitionRegionWithRedundancyTest.putgetWithSingleHop
> * ···
> Also, this is normally the exec flow for the TCs that get stuck:
> # Setup cluster
> # Do TC specific ops
> # Stop server(s)/Cluster shutdown
> # Start server(s)
> In all cases, the server gets stuck at step 4
--
This message was sent by Atlassian Jira
(v8.20.1#820001)