[jira] [Created] (IMPALA-7773) ScanRange::ReadFromCache() holds RequestContext lock while opening file

["Joe McDonnell (JIRA)" <ji...@apache.org>]

Joe McDonnell created IMPALA-7773:
-------------------------------------

             Summary: ScanRange::ReadFromCache() holds RequestContext lock while opening file
                 Key: IMPALA-7773
                 URL: https://issues.apache.org/jira/browse/IMPALA-7773
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 3.1.0
            Reporter: Joe McDonnell


When a scanner thread is reading a file via HDFS caching, it executes ScanRange::ReadFromCache() while holding the RequestContext::lock_. ReadFromCache() calls Open(), which can require an RPC to the NameNode for HDFS. If the NameNode is slow (or hangs), this will be holding the lock for the duration. This lock is used by other scanner threads for this scan node. Disk Io Mgr threads sometimes need to get this lock (e.g. RequestContext::ReadDone(), called by DiskThreadLoop()). This can severely impact the system.

We should look into what it would take to drop the lock during the Open() call (and during the subsequent hadoopReadZero() call). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org