[jira] [Commented] (HDFS-6912) SharedFileDescriptorFactory should not allocate sparse files

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111107#comment-14111107 ] 

Todd Lipcon commented on HDFS-6912:
-----------------------------------

Hey Colin. Did you verify that tmpfs supports fallocate going back to old versions? Looking at the kernel git history, it was only added in mid 2012 (e2d12e22c59ce714008aa5266d769f8568d74eac) corresponding to version 3.5. So, I'm not sure if it would be supported on el6 for example (maybe they backported it, maybe not).

Doing a normal posix write() call to write some explicit zeros to the fd might be more portable and shouldn't really have any performance downside.

> SharedFileDescriptorFactory should not allocate sparse files
> ------------------------------------------------------------
>
>                 Key: HDFS-6912
>                 URL: https://issues.apache.org/jira/browse/HDFS-6912
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: caching
>    Affects Versions: 2.5.0
>         Environment: HDFS Data node, with 8 gb tmpfs in /dev/shm
>            Reporter: Gopal V
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-6912.001.patch
>
>
> SharedFileDescriptor factory should not allocate sparse files.  Sparse files can lead to a SIGBUS later in the short-circuit reader when we try to read from the sparse file and memory is not available.
> Note that if swap is enabled, we can still get a SIGBUS even with a non-sparse file, since the JVM uses MAP_NORESERVE in mmap.



--
This message was sent by Atlassian JIRA
(v6.2#6252)