[jira] [Commented] (ACCUMULO-4080) TabletServers should be less aggressively "monitoring RO filesystems"

["Josh Elser (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062454#comment-15062454 ] 

Josh Elser commented on ACCUMULO-4080:
--------------------------------------

The current mounts on a node I saw this:

{noformat}
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8119336k,nr_inodes=2029834,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/vda1 / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=32,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
sunrpc /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/vdb /grid/0 ext4 rw,seclabel,relatime,data=ordered 0 0
{noformat}

> TabletServers should be less aggressively "monitoring RO filesystems"
> ---------------------------------------------------------------------
>
>                 Key: ACCUMULO-4080
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4080
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.6.4, 1.7.0
>         Environment: {{uname -a}}
> Linux os-r7-ncibzu-ambari-hv-r-upg-4-re1-4 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> {{cat /etc/redhat-release}}
> CentOS Linux release 7.0.1406 (Core)
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.6.5, 1.7.1, 1.8.0
>
>
> Ran into an automated test case where all of the tservers killed themselves on Centos7.
> {noformat}
> 2015-12-17 14:51:30,164 [util.FileSystemMonitor] FATAL: Exception while checking mount points, halting process
> java.lang.Exception: Filesystem /sys/fs/cgroup switched to read only
>         at org.apache.accumulo.server.util.FileSystemMonitor.checkMounts(FileSystemMonitor.java:123)
>         at org.apache.accumulo.server.util.FileSystemMonitor$1.run(FileSystemMonitor.java:90)
>         at java.util.TimerThread.mainLoop(Timer.java:555)
>         at java.util.TimerThread.run(Timer.java:505)
> {noformat}
> I'm not quite sure what exactly happened that caused /sys/fs/cgroup to suddenly be mounted as ro (my hunch is that it was an updated package).
> A workaround is to set {{tserver.monitor.fs}} to {{false}} in accumulo-site.xml and restart Accumulo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)