[jira] [Commented] (HBASE-20280) Fix possibility of deadlocking in refreshFileConnections when prefetch is enabled

["Zach York (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412460#comment-16412460 ] 

Zach York commented on HBASE-20280:
-----------------------------------

[~apurtell] Could you make sure I am understanding the prefetch logic correctly (especially around canceling)? Trying to understand the code, it looks like when a ClosedByInterruptException is thrown it first sets the thread to interrupted then throws the exception.

From [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java#L270] it looks like after the exception is thrown (and caught by BucketCache) this should break out of the loop. However, I saw two exceptions being thrown back to back by apparently the same thread (by name at least) which would indicate that this didn't actually break out of the loop. Do you have any idea what is going on here:

 

2018-03-17 12:03:31,986 ERROR [hfile-prefetch-1521244956862] bucket.BucketCache: Failed reading block f1512d0ed56546939771d8e7f5e72c6a_129839 from bucket cache java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$FileReadAccessor.access(FileIOEngine.java:276) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.accessFile(FileIOEngine.java:196) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:113) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:506) at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:84) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:279) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:420) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(HFileReaderV2.java:209) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-17 12:03:31,986 ERROR [hfile-prefetch-1521244956862] bucket.BucketCache: Failed reading block f1512d0ed56546939771d8e7f5e72c6a_129839 from bucket cache java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine$FileReadAccessor.access(FileIOEngine.java:276) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.accessFile(FileIOEngine.java:196) at org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:113) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:506) at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:84) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:279) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:420) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$1.run(HFileReaderV2.java:209) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

> Fix possibility of deadlocking in refreshFileConnections when prefetch is enabled
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-20280
>                 URL: https://issues.apache.org/jira/browse/HBASE-20280
>             Project: HBase
>          Issue Type: Bug
>          Components: BucketCache
>    Affects Versions: 2.0.0-beta-2, 1.4.2
>            Reporter: Zach York
>            Assignee: Zach York
>            Priority: Major
>         Attachments: HBASE-20280.master.001.patch
>
>
> When prefetch on open is specified, there is a deadlocking case
> where if the prefetch is cancelled, the PrefetchExecutor interrupts
> the threads if necessary, when that happens in FileIOEngine, it
> causes an ClosedByInterruptException which is a subclass of
> ClosedChannelException. If we retry all ClosedChannelExceptions,
> this will lock as this access is expected to be interrupted.
> This change removes calling refreshFileConnections for
> ClosedByInterruptExceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)