[jira] [Created] (HDFS-7272) Concurrency problem in Dfsck/URLConnectionFactory classes

["Maciej Arciuch (JIRA)" <ji...@apache.org>]

Maciej Arciuch created HDFS-7272:
------------------------------------

             Summary: Concurrency problem in Dfsck/URLConnectionFactory classes
                 Key: HDFS-7272
                 URL: https://issues.apache.org/jira/browse/HDFS-7272
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.3.0
            Reporter: Maciej Arciuch
            Priority: Minor


Hi,

I think I've found a concurrency issue in the URLConnectionFactory class used by the Dfsck class. 

The problem is that when multiple instances of new Dfsck run in the same JVM, 401 errors occur. I know that this class in usually run in different command-line processes, but if I call programatically Dfsck.doWork() on multiple instances in parallel I get strange, random auth errors. Earlier versions worked fine.

So, I took a look at the code and that's I found:

In the earlier versions of the Dfsck the connection was instatiated using SecurityUtil.openSecureHttpConnection(path), which worked just fine. New versions use URLConnectionFactory.

Quick links to grepcode:

old, correct code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.0.0-cdh4.2.1/org/apache/hadoop/hdfs/tools/DFSck.java/#161

new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/tools/DFSck.java/#174

The difference is that the old SecurityUtil always passed null Authenticator to the AuthenticatedURL constructor, which, if case of null argument, instantiated a new Authenticator instance using the default instantiator class. *So, there was new Authenticator instance created for each call. The URLConnectorFactory passes a non-null, static Authenticator, which is stateful and not thread-safe.*

old code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-common/2.0.0-cdh4.2.1/org/apache/hadoop/security/SecurityUtil.java#508

new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/web/URLConnectionFactory.java#164

Can anyone confirm and perhaps fix this? Best regards, Maciej



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)