HDFS timeout setting

[Nasron Cheong <na...@upsight.com>]

We encountered an issue where our hdfs cluster misbehaved, which resulted
in very slow reads. However, existing impala queries would continue even if
cancelled, presumably being blocked by the hdfs read.

Are there settings to be less aggressive on hdfs reads so we can fail more
quickly if hdfs is the issue?

Thanks.

- Nasron

Re: HDFS timeout setting

[Nasron Cheong <na...@upsight.com>]

Thanks, will try.

- Nasron

On Sat, Sep 9, 2017 at 2:41 AM, Jeszy <je...@gmail.com> wrote:

> Hello Nasron,
>
> The read hanging is an HDFS timeout issue, to work around it you would
> need an HDFS version which has HADOOP-12672. After that, you should be
> able to set ipc.client.rpc-timeout.ms to a reasonable value in
> Impala's HDFS client settings.
>
> HTH
>
> On 8 September 2017 at 20:24, Nasron Cheong <na...@upsight.com>
> wrote:
> > We encountered an issue where our hdfs cluster misbehaved, which
> resulted in
> > very slow reads. However, existing impala queries would continue even if
> > cancelled, presumably being blocked by the hdfs read.
> >
> > Are there settings to be less aggressive on hdfs reads so we can fail
> more
> > quickly if hdfs is the issue?
> >
> > Thanks.
> >
> > - Nasron
>

Re: HDFS timeout setting

[Jeszy <je...@gmail.com>]

Hello Nasron,

The read hanging is an HDFS timeout issue, to work around it you would
need an HDFS version which has HADOOP-12672. After that, you should be
able to set ipc.client.rpc-timeout.ms to a reasonable value in
Impala's HDFS client settings.

HTH

On 8 September 2017 at 20:24, Nasron Cheong <na...@upsight.com> wrote:
> We encountered an issue where our hdfs cluster misbehaved, which resulted in
> very slow reads. However, existing impala queries would continue even if
> cancelled, presumably being blocked by the hdfs read.
>
> Are there settings to be less aggressive on hdfs reads so we can fail more
> quickly if hdfs is the issue?
>
> Thanks.
>
> - Nasron