You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2012/08/02 19:28:03 UTC
[jira] [Created] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Eli Collins created HDFS-3754:
---------------------------------
Summary: BlockSender doesn't shutdown ReadaheadPool threads
Key: HDFS-3754
URL: https://issues.apache.org/jira/browse/HDFS-3754
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 2.0.0-alpha, 1.1.0
Reporter: Eli Collins
Assignee: Eli Collins
The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Right patch this time.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428985#comment-13428985 ]
Todd Lipcon commented on HDFS-3754:
-----------------------------------
I'm a little confused, doesn't this have the same problem as before for tests? i.e one DN shutting down will call ReadaheadPool.shutdown() on the only instance, but anyone else who keeps a reference to that one will get exceptions trying to submit to the executor?
Maybe we could fix this more easily by just changing the ReadaheadPool ThreadFactory so that the threads don't get created inside any thread group? Then they'd exit on JVM exit properly.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428771#comment-13428771 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539170/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2957//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2957//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3754.
-------------------------------
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 1.2.0, 2.0.2-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754-b1.txt
Updated branch-1 patch. The fix for branch-1 also requires that the invocation of Executors#defaultThreadFactory be hoisted to the constructor, this is because the ThreadGroup used is that of the thread invoking defaultThreadFactory and we want that to be the primary thread group rather than the thread group of whatever thread causes the pool to create a new thread. Trunk doesn't have this problem because it uses guava's ThreadFactoryBuilder which behaves similarly (does not invoke defaultThreadFactory on each newThread call). Running {{ant test -Dtestcase=TestSeekBug -Dcompile.native=true}} no longer hangs with this patch.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Arg, you're right. Updated patch which takes option c attached.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431095#comment-13431095 ]
Hudson commented on HDFS-3754:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1129 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1129/])
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. Contributed by Eli Collins (Revision 1370495)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Updated patch with comment per Colin's suggestion.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427553#comment-13427553 ]
Todd Lipcon commented on HDFS-3754:
-----------------------------------
This doesn't look right. The ReadaheadPool is static. To fix this, I think you need to either (a) move the ReadaheadPool inside the DN, and shut it down from the DN context, (b) change the classloading of BlockSender so that the ReadaheadPool doesn't get created inside the Datanode ThreadGroup, or (c) have the DN shutdown method call something like {{BlockSender.readaheadPool.shutdown()}}
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427714#comment-13427714 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538946/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper
org.apache.hadoop.hdfs.TestShortCircuitLocalRead
org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.TestFSInputChecker
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDataTransferProtocol
org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
org.apache.hadoop.hdfs.TestDFSClientFailover
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
org.apache.hadoop.fs.permission.TestStickyBit
org.apache.hadoop.hdfs.TestFileConcurrentReader
org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
org.apache.hadoop.fs.TestFcHdfsCreateMkdir
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.TestAppendDifferentChecksum
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
org.apache.hadoop.hdfs.TestDFSPermission
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
org.apache.hadoop.hdfs.TestListFilesInFileContext
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.TestDatanodeReport
org.apache.hadoop.hdfs.TestDFSRemove
org.apache.hadoop.hdfs.TestRestartDFS
org.apache.hadoop.hdfs.TestFSOutputSummer
org.apache.hadoop.hdfs.TestDatanodeRegistration
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
org.apache.hadoop.hdfs.TestHDFSTrash
org.apache.hadoop.hdfs.TestDFSRollback
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestClientReportBadBlock
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
org.apache.hadoop.hdfs.TestBalancerBandwidth
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.server.datanode.TestHSync
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
org.apache.hadoop.hdfs.TestListPathServlet
org.apache.hadoop.hdfs.server.namenode.TestStreamFile
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
org.apache.hadoop.hdfs.TestFileCreationDelete
org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.TestFileAppend
org.apache.hadoop.hdfs.TestMissingBlocksAlert
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
org.apache.hadoop.hdfs.TestSeekBug
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
org.apache.hadoop.hdfs.TestHDFSServerPorts
org.apache.hadoop.hdfs.TestAbandonBlock
org.apache.hadoop.hdfs.server.datanode.TestDiskError
org.apache.hadoop.hdfs.TestRenameWhileOpen
org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
org.apache.hadoop.hdfs.TestBlockReaderLocal
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
org.apache.hadoop.hdfs.TestGetBlocks
org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
org.apache.hadoop.fs.viewfs.TestViewFsHdfs
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.TestBlockMissingException
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.TestQuota
org.apache.hadoop.fs.TestUrlStreamHandler
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
org.apache.hadoop.fs.TestHDFSFileContextMainOperations
org.apache.hadoop.hdfs.TestSetrepDecreasing
org.apache.hadoop.hdfs.TestFileCreationEmpty
org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.TestDatanodeDeath
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.TestFileStatus
org.apache.hadoop.hdfs.TestFileAppend3
org.apache.hadoop.hdfs.TestHFlush
org.apache.hadoop.hdfs.TestIsMethodSupported
org.apache.hadoop.hdfs.TestListFilesInDFS
org.apache.hadoop.hdfs.TestPipelines
org.apache.hadoop.security.TestRefreshUserMappings
org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp
org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
org.apache.hadoop.fs.TestGlobPaths
org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
org.apache.hadoop.hdfs.server.namenode.TestMetaSave
org.apache.hadoop.hdfs.TestDFSStartupVersions
org.apache.hadoop.hdfs.TestFileCreationClient
org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.journalservice.TestJournalService
org.apache.hadoop.hdfs.TestLeaseRecovery
org.apache.hadoop.hdfs.TestLocalDFS
org.apache.hadoop.security.TestPermission
org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp
org.apache.hadoop.fs.TestFcHdfsSymlink
org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
org.apache.hadoop.hdfs.TestDFSRename
org.apache.hadoop.hdfs.TestHftpFileSystem
org.apache.hadoop.hdfs.TestDFSAddressConfig
org.apache.hadoop.hdfs.TestFetchImage
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart
org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.TestLeaseRecovery2
org.apache.hadoop.hdfs.TestWriteRead
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.TestMiniDFSCluster
org.apache.hadoop.hdfs.TestFileCorruption
org.apache.hadoop.hdfs.TestHDFSFileSystemContract
org.apache.hadoop.hdfs.TestReadWhileWriting
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
org.apache.hadoop.hdfs.TestDFSFinalize
org.apache.hadoop.hdfs.TestMultiThreadedHflush
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
org.apache.hadoop.hdfs.TestDFSUpgrade
org.apache.hadoop.hdfs.TestDFSMkdirs
org.apache.hadoop.hdfs.TestDatanodeConfig
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestConnCache
org.apache.hadoop.fs.TestResolveHdfsSymlink
org.apache.hadoop.hdfs.TestDFSShellGenericOptions
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend
org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot
org.apache.hadoop.hdfs.TestModTime
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.TestDataTransferKeepalive
org.apache.hadoop.hdfs.TestLargeBlock
org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
org.apache.hadoop.hdfs.TestClientBlockVerification
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
org.apache.hadoop.hdfs.TestParallelLocalRead
org.apache.hadoop.hdfs.TestSetTimes
org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestPersistBlocks
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.fs.TestFcHdfsPermission
org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.TestFileAppendRestart
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2939//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2939//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429665#comment-13429665 ]
Aaron T. Myers commented on HDFS-3754:
--------------------------------------
The specific test case which failed was recently re-enabled after having been disabled for a very long time. Quite possible the failure is unrelated to this particular patch, but worth looking in to.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429635#comment-13429635 ]
Todd Lipcon commented on HDFS-3754:
-----------------------------------
+1
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430673#comment-13430673 ]
Hudson commented on HDFS-3754:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2580 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2580/])
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. Contributed by Eli Collins (Revision 1370495)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427824#comment-13427824 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3754:
----------------------------------------------
BlockSender readaheadPool is static. If we shutdown it, could we still able to restart DNs or shutdown some of the DNs when running multiple DNs in unit tests?
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429554#comment-13429554 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
Colin, thanks, sure, I'll add a comment.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429552#comment-13429552 ]
Colin Patrick McCabe commented on HDFS-3754:
--------------------------------------------
Initializing the ReadaheadPool in DataNode seems like a good idea to me. I also tested the latest patch, and it worked for me.
Would it be worthwhile to add a comment to ReadaheadPool about the importance of having the correct thread context? Or maybe just a reference to this JIRA. I know that I definitely wouldn't have considered the importance of thread context in this situation.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Good point, it's possible in a test that an existing active block sender could race with the shutdown of another DN and submit to the pool that's been shutdown.
I like the idea making the ReadaheadPool pool not part of the dataXceiverServer thread group, this can actually be accomplished more easily by just moving the initialization from BlockSender to DataNode, which is a more logical place anyway. Updated patch attached.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430637#comment-13430637 ]
Hudson commented on HDFS-3754:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2626 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2626/])
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. Contributed by Eli Collins (Revision 1370495)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427552#comment-13427552 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
No tests since this unblocks the existing tests.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427477#comment-13427477 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
(this test only hangs btw if HDFS-3697 is also applied)
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754-b1.txt
Thanks, here's the patch for branch-1.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-3754:
----------------------------------
Affects Version/s: (was: 1.1.0)
1.2.0
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427544#comment-13427544 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538927/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2938//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2938//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Patch attached. I confirmed TestSeekBug no longer hangs (after using the workaround from HDFS-3753) with this patch.
The same change applies to branch-1, verified {{ant clean test -Dtestcase=TestSeekBug -Dcompile.native=true}} no longer hangs.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Target Version/s: 1.2.0 (was: 1.2.0, 2.2.0-alpha)
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
I've committed this to trunk and merged to branch-2.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429680#comment-13429680 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
Yea, was looking at that. Don't think it's related filed HDFS-3770 with the rationale. Sanity checked that this test passes for me w/ this patch applied for a couple of runs.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427487#comment-13427487 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
This affects trunk, branch-2 and branch-1.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430837#comment-13430837 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
Forgot to mention, in this case DataXceiverServer is creating the threads and it has the ThreadGroup that the DN is checking the count against which is why we don't want threads created with this ThreadGroup, and logically the ReadaHead thread pool doesn't need to share a group with the dataXceiverServer threads.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Status: Patch Available (was: Open)
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 2.0.0-alpha, 1.1.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427821#comment-13427821 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538979/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2941//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2941//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429611#comment-13429611 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539360/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2958//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2958//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Status: Open (was: Patch Available)
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 2.0.0-alpha, 1.2.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429640#comment-13429640 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539370/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileConcurrentReader
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2959//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2959//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431121#comment-13431121 ]
Hudson commented on HDFS-3754:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1161 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1161/])
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. Contributed by Eli Collins (Revision 1370495)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Yea, I think moving the ReadaheadPool intialization and shutdown to the DN makes more sense after all. Updated patch attached.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427478#comment-13427478 ]
Suresh Srinivas commented on HDFS-3754:
---------------------------------------
Eli, this affect only 1.2.0 release right. Do we have read ahead changes in 1.1.0?
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427488#comment-13427488 ]
Eli Collins commented on HDFS-3754:
-----------------------------------
The only reason {{mvn -Dtest=TestSeekBug test -Pnative}} doesn't hang on trunk right now is because HDFS-3753 is preventing the native library from being loaded.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.1.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428553#comment-13428553 ]
Hadoop QA commented on HDFS-3754:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539125/hdfs-3754.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2951//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2951//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2951//console
This message is automatically generated.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3754:
------------------------------
Attachment: hdfs-3754.txt
Move the resetting of the static instance member to it's own static method (out of the instance method) to appease jenkins.
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3754) BlockSender doesn't shutdown
ReadaheadPool threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430634#comment-13430634 ]
Hudson commented on HDFS-3754:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2561 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2561/])
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. Contributed by Eli Collins (Revision 1370495)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> BlockSender doesn't shutdown ReadaheadPool threads
> --------------------------------------------------
>
> Key: HDFS-3754
> URL: https://issues.apache.org/jira/browse/HDFS-3754
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 1.2.0, 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Eli Collins
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3754-b1.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt, hdfs-3754.txt
>
>
> The BlockSender doesn't shutdown the ReadaheadPool threads so when tests are run with native libraries some tests fail (time out) because shutdown hangs waiting for the outstanding threads to exit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira