[jira] [Resolved] (HDDS-781) Ambari HDP NoClassDefFoundError for MR jobs

["Arpit Agarwal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arpit Agarwal resolved HDDS-781.
--------------------------------
    Resolution: Won't Fix

Ambari HDP is not supported by Apache Ozone.

> Ambari HDP NoClassDefFoundError for MR jobs
> -------------------------------------------
>
>                 Key: HDDS-781
>                 URL: https://issues.apache.org/jira/browse/HDDS-781
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Filesystem
>    Affects Versions: 0.3.0
>            Reporter: Soumitra Sulav
>            Priority: Major
>
> HDP integrated with Ambari has a _/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the libraries needed for a MR job to run and is copied in the yarn containers at time of execution.
> As introducing ozone filesystem, relevant jars need to be packaged as part of the tar, also the tar is placed as part of _yum install hadoop_ components done by Ambari during cluster setup.
> During an MR Job run, I faced below java.lang.*NoClassDefFoundError* exceptions :
> {code:java}
> org/apache/hadoop/fs/ozone/OzoneFileSystem
> org/apache/ratis/proto/RaftProtos$ReplicationLevel
> org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum
> {code}
>  
> Adding the relevant jar in the mentioned tar file resolves the exception.
> Complete stacktrace for one of the *NoClassDefFoundError* exception :
> {code:java}
> 2018-10-31 10:03:05,191 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel
>  at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
>  at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
>  at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
>  at org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
>  at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
>  at org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
>  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
>  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
>  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
>  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
>  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
>  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
>  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
>  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
>  at org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
>  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
>  at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
>  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
>  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
> Caused by: java.lang.ClassNotFoundException: org.apache.ratis.proto.RaftProtos$ReplicationLevel
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 29 more
> 2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org