[GitHub] [accumulo] EdColeman opened a new issue #1598: Transient MasterMetricsIT Jenkins test failure.

[GitBox <gi...@apache.org>]

EdColeman opened a new issue #1598:
URL: https://github.com/apache/accumulo/issues/1598


   Recent Jenkins build failed - it looks like there was resource contention.  The current test spins up multiple compaction (4) and the monitors the metrics for the FATE transactions to start.  It looks like the FATE transactions are queued, but never run.  The 4 transactions were used during development to drive interesting behaviour, but are not strictly necessary for a functional check and the number of simultaneous running transactions can be reduced.
   
   `Compactions launched:
   
   2020-04-27T13:16:46,388` [util.SlowOps] DEBUG: Compaction running for MasterMetricsIT_compactionMetrics0_0
   2020-04-27T13:16:46,388 [util.SlowOps] DEBUG: Compaction running for MasterMetricsIT_compactionMetrics0_1
   2020-04-27T13:16:46,388 [util.SlowOps] DEBUG: Compaction running for MasterMetricsIT_compactionMetrics0_2
   2020-04-27T13:16:49,393 [util.SlowOps] DEBUG: Compaction running for MasterMetricsIT_compactionMetrics0_3
   2020-04-27T13:16:49,393 [functional.MasterMetricsIT] INFO : Received metrics LineUpdate{lastUpdate=26017212078, line='1587993403681 master.fate: Context=master, ProcessName=Master, Hostname=jenkins.revelc.net, currentFateOps=0, totalFateOps=2, totalZkConnErrors=0, FateTxState_NEW=0, FateTxState_IN_PROGRESS=0, FateTxState_FAILED_IN_PROGRESS=0, FateTxState_FAILED=0, FateTxState_SUCCESSFUL=0, FateTxState_UNKNOWN=0'}
   
   When the test timed out, the FATEs still had not appeared:
   
   2020-04-27T13:20:04,412 [functional.MasterMetricsIT] INFO : Received metrics update LineUpdate{lastUpdate=221069792248, line='1587993598681 master.fate: Context=master, ProcessName=Master, Hostname=jenkins.revelc.net, currentFateOps=0, totalFateOps=2, totalZkConnErrors=0, FateTxState_NEW=0, FateTxState_IN_PROGRESS=0, FateTxState_FAILED_IN_PROGRESS=0, FateTxState_FAILED=0, FateTxState_SUCCESSFUL=0, FateTxState_UNKNOWN=0'}
   2020-04-27T13:20:11,758 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : Unexpectedly had 4 task(s) remaining in threadpool for execution when being stopped
   `
   
   The timestamps in the metrics show the metrics are updating.  There was some FATE activity that occurred - however, the running compactions never appeared - likely due to resource contention.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org