[jira] [Updated] (HDFS-5461) fallback to non-ssr(local short circuit reads) while oom detected

["Liang Xie (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Liang Xie updated HDFS-5461:
----------------------------

    Attachment: HDFS-5461.txt

bq.  It's because each open stream holds a buffer, and we have hundreds of open streams?
i am not 100% sure, but in my mind, i agree with you,  this oom is easy to repro while we have lots of opened storefiles to be read(e.g. compaction can't catch up sometimes)

Oh, i see, seems the fallback only meaningful for some config like mine:  big Xmx and small MaxDirectMemorySize :)

I attached a patch with more logging about using/pooled direct buffer size. In my option, it could be useful probably while online resetting the log level to "trace"  during OOM occur.  And add a simple try/catch fallback handle for OOM without introducing any config value, per me, seems this way is more reasonable:)

> fallback to non-ssr(local short circuit reads) while oom detected
> -----------------------------------------------------------------
>
>                 Key: HDFS-5461
>                 URL: https://issues.apache.org/jira/browse/HDFS-5461
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.0.0, 2.2.0
>            Reporter: Liang Xie
>         Attachments: HDFS-5461.txt
>
>
> Currently, the DirectBufferPool used by ssr feature seems doesn't have a upper-bound limit except DirectMemory VM option. So there's a risk to encounter direct memory oom. see HBASE-8143 for example.
> IMHO, maybe we could improve it a bit:
> 1) detect OOM or reach a setting up-limit from caller, then fallback to non-ssr
> 2) add a new metric about current raw consumed direct memory size.



--
This message was sent by Atlassian JIRA
(v6.1#6144)