You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Rakesh R (JIRA)" <ji...@apache.org> on 2016/07/04 17:40:10 UTC
[jira] [Commented] (HDFS-10592) Fix intermittent test failure of
TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
[ https://issues.apache.org/jira/browse/HDFS-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361632#comment-15361632 ]
Rakesh R commented on HDFS-10592:
---------------------------------
As per the analysis the test case failure is due to the {{ConcurrentModificationException}} while iterating over the mockito stubs.
Following are the two possible concurrent operations which results in the above exception.
operation-1) NameNodeResourceMonitor is running and invoked {{FSNamesystem.checkAvailableResources}}
operation-2) At the same time, adding the answer {{Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false);}} to the {{InvocationContainerImpl#stubbed}} linked list data structure.
Reference log:
{code}
2016-07-04 12:00:05,657 [DataNode: [[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/, [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:46553] INFO datanode.DataNode (BPOfferService.java:processCommandFromActive(696)) - Got finalize command for block pool BP-1174120436-172.17.0.2-1467633603414
2016-07-04 12:00:05,757 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active
2016-07-04 12:00:05,760 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1] ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in NameNodeResourceMonitor:
java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71)
at org.mockito.internal.MockHandler.handle(MockHandler.java:93)
at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791)
at java.lang.Thread.run(Thread.java:745)
2016-07-04 12:01:05,772 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster
2016-07-04 12:01:05,773 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(1947)) - Shutting down DataNode 0
{code}
org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not thread safe.
{code}
org.mockito.internal.stubbing.InvocationContainerImpl.java
public StubbedInvocationMatcher findAnswerFor(Invocation invocation) {
for (StubbedInvocationMatcher s : stubbed) {
if (s.matches(invocation)) {
s.markStubUsed(invocation);
invocation.markStubbed(new StubInfo(s));
return s;
}
}
return null;
}
{code}
*Proposed Fix:*
Instead of using {{Mockito}}, it is pretty simple to use {{NameNodeResourceCheckerStub}} stubbing and sets the resource availability explicitly. I will attach a patch soon.
> Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10592
> URL: https://issues.apache.org/jira/browse/HDFS-10592
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Rakesh R
> Assignee: Rakesh R
>
> This jira is to fix the {{TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning}} test case failure.
> Reference [Build_15973|https://builds.apache.org/job/PreCommit-HDFS-Build/15973/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNameNodeResourceChecker/testCheckThatNameNodeResourceMonitorIsRunning/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org