Help needed: failed unit test based on miniMRCluster on Yarn with cdh 5.1.3

[Lumin Zhang <lu...@adobe.com>]

Hi, folks,

I have encountered a strange error when I am trying to upgrade from cdh4 to cdh5.1.3.

I have a unit test based on miniMRcluster and ClusterMapReduceTestCase. It runs successfully with cdh 4. After I have changed my maven setting to upgrade to cdh5. The running always failed.

I have tried to isolate the problem. So I used the standard unit test (https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java)  and compile it with my maven setting, then I reproduced the same error by only testing testRestart().


22596 [main] INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory - Stopping JobHistory

22596 [main] INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory - Stopping History Cleaner/Move To Done

22597 [Thread[Thread-184,5,main]] ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted

22597 [main] INFO org.apache.hadoop.hdfs.MiniDFSCluster - Shutting down the Mini HDFS Cluster

22597 [main] INFO org.apache.hadoop.hdfs.MiniDFSCluster - Shutting down DataNode 1

22598 [main] WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner - DirectoryScanner: shutdown has been called


The test should be compatible for both MR1 and MR2.  I also tried to overwrite some Yarn configuration, but it didn't help. I tried to google the related issue, but no luck for this particular issue.Any comments or Help are appreciated. Thanks so much!


My mvn configuration part is as follows,


    <version.hadoop>2.3.0-cdh5.1.3</version.hadoop>

    <version.hadoop.core>2.3.0-cdh5.1.3</version.hadoop.core>

    <version.hadoop.minicluster>2.3.0-cdh5.1.3</version.hadoop.minicluster>


    <dependency>

      <groupId>org.apache.hadoop</groupId>

      <artifactId>hadoop-client</artifactId>

      <version>${version.hadoop}</version>

    </dependency>


    <dependency>

      <groupId>org.apache.hadoop</groupId>

      <artifactId>hadoop-minicluster</artifactId>

      <version>${version.hadoop.minicluster}</version>

      <exclusions>

        <exclusion>

          <groupId>org.slf4j</groupId>

          <artifactId>slf4j-log4j12</artifactId>

        </exclusion>

      </exclusions>

    </dependency>


    <dependency>

      <groupId>${project.groupId}</groupId>

      <artifactId>common-hadoop</artifactId>

      <version>${project.version}</version>

    </dependency>




Best regards,

Lumin Zhang