Fwd: FileSystem object's close method is not called. hadoop 2.7.2

[M G <mg...@gmail.com>]

Hi, I have implemented an HC file system by extending the
org.apache.hadoop.fs.FileSystem class.
Things seem to work fine, except the fact that when I run mapreduce jobs
with yarn (simple DFSIO tests) I keep noticing that my Filesystem's "close"
method is not called.
That is, I see in logs that my file system object is instantiated seven
times during the run of the MRAppMaster process, but the close method was
never closed for any instantiation.
I know that there is a configuration key fs.automatic.close that is checked
on MRappMaster.java - and I make sure that this key is true.
I am using hadoop 2.7.2
Other processes that instantiate my FS class (node manager for example) are
calling the close method, I saw the problem only with MRAppMaster's process.
The DFSIO test succeeds, but this issue can lead to some resources leakage
so it's important that the close method is called.

Thanks,
Michael

Re: FileSystem object's close method is not called. hadoop 2.7.2

[Chris Nauroth <cn...@hortonworks.com>]

Hello Michael,

Historically, there has never been a firm requirement that clients must call FileSystem#close upon finishing usage of an instance.  I think the history here is that the close method was not part of the initial API definition, and when it was added, there were already a lot of existing applications coded that were not calling the close method.  A tactical decision was made not to force all of those applications to make code changes, so calling close has been treated as optional.  (This all pre-dates my time on the project though, so I might not have the story completely right.)

I'd prefer a hard requirement that applications must call close, but this is our current situation.  If you see a specific point in the codebase where you'd like to propose adding a close call, then I recommend filing an Apache JIRA to discuss the proposal.  If you'd like, also consider attaching a patch to the JIRA.

--Chris Nauroth

From: M G <mg...@gmail.com>>
Date: Wednesday, July 13, 2016 at 7:23 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Fwd: FileSystem object's close method is not called. hadoop 2.7.2

Hi, I have implemented an HC file system by extending the org.apache.hadoop.fs.FileSystem class.
Things seem to work fine, except the fact that when I run mapreduce jobs with yarn (simple DFSIO tests) I keep noticing that my Filesystem's "close" method is not called.
That is, I see in logs that my file system object is instantiated seven times during the run of the MRAppMaster process, but the close method was never closed for any instantiation.
I know that there is a configuration key fs.automatic.close that is checked on MRappMaster.java - and I make sure that this key is true.
I am using hadoop 2.7.2
Other processes that instantiate my FS class (node manager for example) are calling the close method, I saw the problem only with MRAppMaster's process.
The DFSIO test succeeds, but this issue can lead to some resources leakage so it's important that the close method is called.

Thanks,
Michael