[GitHub] [hbase] joshelser commented on issue #1029: HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad

[GitBox <gi...@apache.org>]

joshelser commented on issue #1029: HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad
URL: https://github.com/apache/hbase/pull/1029#issuecomment-573781272
 
 
   The problem with #1019 was that other code called beneath SecureBulkLoadManager would expect that there's a cached FileSystem object for it to use. While this is good for us to go through and not willy-nilly expect there to be a cached FS instance hanging around, it does complicate a commit which was intending to simplify things.
   
   Thus, I think it's better for us to go the easy-route, fixing SecureBulkLoadManager.
   
   I've been testing this by letting two instances of IntegrationTestBulkLoad run concurrently with a small chainLength (to let more bulk loads happen), looking at the output of `jmap histo:live` to see how many `DistributedFileSystem` instances we have strongly referenced (it should never be higher than the 3 FS instances we have open, plus the number of concurrent bulk loads, 2, we could have running). It's been running for about an hour; I'll let it continue until QA comes back, or until my HDFS fills up.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services