[jira] [Created] (IMPALA-8237) Enabling preads always fetches hedged reads metrics

["Sahil Takiar (JIRA)" <ji...@apache.org>]

Sahil Takiar created IMPALA-8237:
------------------------------------

             Summary: Enabling preads always fetches hedged reads metrics
                 Key: IMPALA-8237
                 URL: https://issues.apache.org/jira/browse/IMPALA-8237
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


In {{HdfsFileReader}} if preads are enabled, we assume that hedged reads are enabled as well, so whenever we close a file we make a libhdfs call to collect a few hedged read metrics from the underlying {{FileSystem}} object. However, as part of IMPALA-5212 we may want to enable preads even when hedged reads are disabled, so making the call to libhdfs to fetch hedged read metrics will be a waste.

Digging through the HDFS code, it seems the HDFS client triggers hedged reads only if {{dfs.client.hedged.read.threadpool.size}} is greater than 0. We can use the same check in {{HdfsFileReader}} to trigger the fetch of hedged read metrics. The issue is that currently libhdfs does not provide a good way of getting the value of {{dfs.client.hedged.read.threadpool.size}}, it provides a method called {{hdfsConfGetInt}}, but that method simply calls {{new Configuration()}} and fetches the value of  {{dfs.client.hedged.read.threadpool.size}} from it. The issue is that calling {{new Configuration}} simply loads the current {{hdfs-site.xml}}, {{core-site.xml}}, etc. which does not take into account the scenario where the default configuration has been modified for specific filesystem objects - e.g. using {{hdfsBuilder}} to set non-default configuration parameters (see HDFS-14301 for more details).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)