You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2014/12/28 12:41:13 UTC
[jira] [Commented] (HADOOP-11209) Configuration is not thread-safe
[ https://issues.apache.org/jira/browse/HADOOP-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259624#comment-14259624 ]
Hadoop QA commented on HADOOP-11209:
------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12689253/HADOOP-11209.001.patch
against trunk revision 1454efe.
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestDecayRpcScheduler
org.apache.hadoop.http.TestGlobalFilter
org.apache.hadoop.http.TestPathFilter
org.apache.hadoop.fs.viewfs.TestViewFileSystemDelegation
org.apache.hadoop.io.compress.TestCompressionStreamReuse
org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs
org.apache.hadoop.ipc.TestRPCCompatibility
org.apache.hadoop.fs.viewfs.TestViewFsURIs
org.apache.hadoop.fs.shell.TestCopy
org.apache.hadoop.ha.TestHealthMonitor
org.apache.hadoop.ha.TestFailoverController
org.apache.hadoop.security.TestGroupsCaching
org.apache.hadoop.fs.permission.TestFsPermission
org.apache.hadoop.http.TestHttpServer
org.apache.hadoop.security.TestAuthenticationFilter
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
org.apache.hadoop.crypto.key.TestKeyProviderFactory
org.apache.hadoop.conf.TestDeprecatedKeys
org.apache.hadoop.security.authorize.TestProxyUsers
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager
org.apache.hadoop.conf.TestReconfiguration
org.apache.hadoop.net.TestStaticMapping
org.apache.hadoop.fs.TestDU
org.apache.hadoop.fs.contract.localfs.TestLocalFSContractSeek
org.apache.hadoop.fs.TestTruncatedInputBug
org.apache.hadoop.util.TestGenericOptionsParser
org.apache.hadoop.crypto.key.TestKeyShell
org.apache.hadoop.io.TestWritable
org.apache.hadoop.fs.viewfs.TestViewfsFileStatus
org.apache.hadoop.fs.TestTrash
org.apache.hadoop.ha.TestSshFenceByTcpPort
org.apache.hadoop.security.authorize.TestProxyServers
org.apache.hadoop.security.alias.TestCredShell
org.apache.hadoop.ipc.TestIdentityProviders
org.apache.hadoop.ipc.TestIPCServerResponder
org.apache.hadoop.ha.TestShellCommandFencer
org.apache.hadoop.fs.shell.TestCopyPreserveFlag
org.apache.hadoop.cli.TestCLI
org.apache.hadoop.http.TestSSLHttpServer
org.apache.hadoop.fs.shell.find.TestName
org.apache.hadoop.io.compress.zlib.TestZlibCompressorDecompressor
org.apache.hadoop.fs.contract.localfs.TestLocalFSContractOpen
org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS
org.apache.hadoop.fs.contract.ftp.TestFTPContractOpen
org.apache.hadoop.fs.shell.TestCount
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.fs.viewfs.TestFcCreateMkdirLocalFs
org.apache.hadoop.http.TestServletFilter
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
org.apache.hadoop.conf.TestConfigurationDeprecation
org.apache.hadoop.fs.TestFilterFileSystem
org.apache.hadoop.security.ssl.TestSSLFactory
org.apache.hadoop.fs.shell.find.TestPrint0
org.apache.hadoop.service.TestServiceLifecycle
org.apache.hadoop.fs.viewfs.TestViewFileSystemDelegationTokenSupport
org.apache.hadoop.fs.shell.TestXAttrCommands
org.apache.hadoop.ipc.TestMiniRPCBenchmark
org.apache.hadoop.fs.shell.find.TestIname
org.apache.hadoop.io.file.tfile.TestTFileSeek
org.apache.hadoop.util.TestGenericsUtil
org.apache.hadoop.io.serializer.TestWritableSerialization
org.apache.hadoop.crypto.key.TestKeyProvider
org.apache.hadoop.ipc.TestRPC
org.apache.hadoop.io.TestGenericWritable
org.apache.hadoop.conf.TestConfServlet
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractSeek
org.apache.hadoop.ipc.TestProtoBufRpc
org.apache.hadoop.security.TestWhitelistBasedResolver
org.apache.hadoop.fs.shell.find.TestFind
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.security.alias.TestCredentialProviderFactory
org.apache.hadoop.ipc.TestIPC
org.apache.hadoop.ipc.TestMultipleProtocolServer
org.apache.hadoop.ipc.TestWeightedRoundRobinMultiplexer
org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
org.apache.hadoop.fs.TestFsShellReturnCode
org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
org.apache.hadoop.fs.shell.find.TestPrint
org.apache.hadoop.net.TestScriptBasedMappingWithDependency
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
org.apache.hadoop.util.hash.TestHash
org.apache.hadoop.ipc.TestSaslRPC
org.apache.hadoop.net.TestSwitchMapping
org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
org.apache.hadoop.http.lib.TestStaticUserWebFilter
org.apache.hadoop.security.TestShellBasedIdMapping
org.apache.hadoop.ipc.TestRPCCallBenchmark
org.apache.hadoop.io.serializer.avro.TestAvroSerialization
org.apache.hadoop.fs.shell.TestMove
org.apache.hadoop.fs.viewfs.TestViewFsConfig
org.apache.hadoop.security.TestCompositeGroupMapping
org.apache.hadoop.security.authorize.TestServiceAuthorization
org.apache.hadoop.net.TestTableMapping
org.apache.hadoop.security.TestDoAsEffectiveUser
org.apache.hadoop.ha.TestZKFailoverControllerStress
org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem
org.apache.hadoop.fs.viewfs.TestChRootedFileSystem
org.apache.hadoop.fs.TestFileContext
org.apache.hadoop.http.TestHttpCookieFlag
org.apache.hadoop.security.TestGroupFallback
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
org.apache.hadoop.net.TestScriptBasedMapping
org.apache.hadoop.fs.viewfs.TestViewFsLocalFs
org.apache.hadoop.fs.TestLocalFileSystem
org.apache.hadoop.io.TestSequenceFileSerialization
org.apache.hadoop.conf.TestConfiguration
org.apache.hadoop.fs.shell.TestAclCommands
org.apache.hadoop.security.TestSecurityUtil
org.apache.hadoop.io.TestDefaultStringifier
org.apache.hadoop.ipc.TestFairCallQueue
org.apache.hadoop.security.TestUGILoginFromKeytab
org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
org.apache.hadoop.io.compress.TestCodec
org.apache.hadoop.io.serializer.TestSerializationFactory
org.apache.hadoop.security.TestLdapGroupsMapping
org.apache.hadoop.io.compress.TestCodecFactory
org.apache.hadoop.ipc.TestServer
The following test timeouts occurred in hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.TestFileSystemCaching
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5349//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5349//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5349//console
This message is automatically generated.
> Configuration is not thread-safe
> --------------------------------
>
> Key: HADOOP-11209
> URL: https://issues.apache.org/jira/browse/HADOOP-11209
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Reporter: Josh Rosen
> Assignee: Varun Saxena
> Attachments: HADOOP-11209.001.patch
>
>
> {{Configuration}} objects are not fully thread-safe, which causes problems in multi-threaded frameworks like Spark that use these configurations to interact with existing Hadoop APIs (such as InputFormats).
> SPARK-2546 is an example of a problem caused by this lack of thread-safety. In that bug, multiple concurrent modifications of the same Configuration (in third-party code) caused an infinite loop because Configuration's internal {{java.util.HashMap}} is not thread-safe.
> One workaround is for our code to clone Configuration objects; unfortunately, this also suffers from thread-safety issues on older Hadoop versions because Configuration's constructor wasn't thread-safe (HADOOP-10456).
> [Looking at a recent version of Configuration.java|https://github.com/apache/hadoop/blob/d989ac04449dc33da5e2c32a7f24d59cc92de536/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L666], it seems that the private {{updatingResource}} HashMap and {{finalParameters}} HashSet fields the only non-thread-safe collections in Configuration (Java's {{Properties}} class is thread-safe), so I don't think that it would be hard to make Configuration fully thread-safe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)