[GitHub] [accumulo] ctubbsii commented on issue #2016: Github QA occasionally hangs while running unit-tests

[GitBox <gi...@apache.org>]

ctubbsii commented on issue #2016:
URL: https://github.com/apache/accumulo/issues/2016#issuecomment-819692920


   I noticed that in these cases, the logs aren't being uploaded. It looks like the timeout is set on entire jobs, so the upload-artifact steps never happen if the timeout is reached in running the Maven build. This can be addressed by moving the timeout-minutes to the steps, rather than under the job. That way, if any individual step times out, the other steps (like uploading log artifacts) still run.
   
   Also, the last thing done in a workflow before the runner shuts down is to kill orphaned processes. I noticed that there may be fewer processes running than expected. Here is an example from a recent timeout:
   
   ```
   2021-04-14T17:17:43.3208229Z Cleaning up orphan processes
   2021-04-14T17:17:43.3656311Z Terminate orphan process: pid (1580) (java)
   2021-04-14T17:17:43.3799021Z Terminate orphan process: pid (18229) (sh)
   2021-04-14T17:17:43.3833188Z Terminate orphan process: pid (18233) (java)
   2021-04-14T17:17:43.3882297Z Terminate orphan process: pid (18253) (java)
   2021-04-14T17:17:43.3938861Z Terminate orphan process: pid (18361) (java)
   2021-04-14T17:17:43.4044764Z Terminate orphan process: pid (18362) (java)
   2021-04-14T17:17:43.4095156Z Terminate orphan process: pid (18475) (java)
   2021-04-14T17:17:43.4151569Z Terminate orphan process: pid (18476) (java)
   ```
   
   I'm not sure which java processes are still running at the end of the job, but it looks like mini isn't starting or shutting down correctly when everything times out and is terminated. It would be useful to get more information in the output about what these processes are, in order to determine what is causing the problem. A simple "ps aux | grep jav[a]" or similar might help provide more insight into which processes are left running (and perhaps which aren't running that should have been).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org