[jira] [Created] (TRAFODION-1051) LP Bug: 1430876 - infra - hadoop/hbase fill up root filesystem

["Alice Chen (JIRA)" <ji...@apache.org>]

Alice Chen created TRAFODION-1051:
-------------------------------------

             Summary: LP Bug: 1430876 - infra - hadoop/hbase fill up root filesystem
                 Key: TRAFODION-1051
                 URL: https://issues.apache.org/jira/browse/TRAFODION-1051
             Project: Apache Trafodion
          Issue Type: Bug
          Components: Build Infrastructure
            Reporter: Steve Varnau
            Assignee: Steve Varnau
            Priority: Blocker


In some circumstances, hadoop and hbase spew gigabytes of messages into log files, filling up root filesystem.  Because jenkins can’t write a temp file, the next job fails completely.  Jenkins eventually detects machine out of disk space and disables it, but it may cause several jobs to fail.

The problem seems to happen pretty rapidly.  Since jenkins uses /tmp, which shares filesystem with /var/log, a job cannot run to clean things up.

Running "hbase-clean.sh initialize" gets hbase/hadoop out of bad mode (once some disk space is reclaimed).

Ideas:
- can we get jenkins to use a different temp directory (/mnt/tmp)?
  Then we could check for and clean up at beginning of a job.
- Run some sort of cron job to clean up /var/log/h*
- Run postbuild step to stop hbase and clean up /var/log?
  That should take care of jobs that finish normally or timeout.  But if disk is already full, that might prevent step from running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)