[jira] [Comment Edited] (PARQUET-783) H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks

["Rohit Aggarwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031811#comment-16031811 ] 

Rohit Aggarwal edited comment on PARQUET-783 at 5/31/17 7:34 PM:
-----------------------------------------------------------------

We've been hit by this same issue as well. Any chance of 1.9.1 happening soon? 

imho this is a quite serious issue given how extensively parquet is used.
 
Thanks! 


was (Author: rohit.aggarwal):
We've been hit by this same issue as well. Any chance of 1.9.1 happening soon? Thanks! 

> H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks
> --------------------------------------------------------------------------------------------------
>
>                 Key: PARQUET-783
>                 URL: https://issues.apache.org/jira/browse/PARQUET-783
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.9.0
>            Reporter: Michael Allman
>            Assignee: Michael Allman
>            Priority: Critical
>             Fix For: 1.10.0
>
>
> {{ParquetFileReader}} opens a {{SeekableInputStream}} to read a footer. In the process, it opens a new {{FSDataInputStream}} and wraps it. However, {{H2SeekableInputStream}} does not override the {{close}} method. Therefore, when {{ParquetFileReader}} closes it, the underlying {{FSDataInputStream}} is not closed. As a result, these stale connections can exhaust a clusters' data nodes' connection resources and lead to mysterious HDFS read failures in HDFS clients, e.g.
> {noformat}
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-905337612-172.16.70.103-1444328960665:blk_1720536852_646811517
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)