You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2022/07/11 08:46:00 UTC

[jira] [Commented] (HDDS-6926) Cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message

    [ https://issues.apache.org/jira/browse/HDDS-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564870#comment-17564870 ] 

Attila Doroszlai commented on HDDS-6926:
----------------------------------------

bq. Does not currently support Shaded Hadoop jars in Ozone?

I think that's exactly the case.

> Cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
> ----------------------------------------------------------------------
>
>                 Key: HDDS-6926
>                 URL: https://issues.apache.org/jira/browse/HDDS-6926
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: OM
>    Affects Versions: 1.2.1
>         Environment: Ozone version:
> 1.2.1 and 1.3.0
> Spark version:
> 3.2.1
>            Reporter: MLikeWater
>            Priority: Major
>
> The test process is as follows:
> 1. Download and use the official Spark installation package:
> {code:java}
> spark-3.2.1-bin-hadoop3.2.tgz
> {code}
> 2. Place the ozone configuration file *ozone-site.xml* in ${SPARK_HOME}/conf and *ozone-filesystem-hadoop3-1.2.1.jar* in ${SPARK_HOME}/jars:
> {code:java}
> $ ll spark-3.2.1-bin-hadoop3.2/jars/hadoop-*
> rw-rr- 1 hadoop hadoop 19406393 Jun 17 11:27 spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> rw-rr- 1 hadoop hadoop 31717292 Jun 17 11:27 spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-runtime-3.3.1.jar
> rw-rr- 1 hadoop hadoop  3362359 Jun 17 11:27 spark-3.2.1-bin-hadoop3.2/jars/hadoop-shaded-guava-1.1.1.jar
> rw-rr- 1 hadoop hadoop    56507 Jun 17 11:27 spark-3.2.1-bin-hadoop3.2/jars/hadoop-yarn-server-web-proxy-3.3.1.jar
> \$ ll spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar 
> rw-rr- 1 hadoop hadoop 61732949 Jun 18 15:15 spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> \$ ll spark-3.2.1-bin-hadoop3.2/conf/ozone-site.xml
> rw-rr- 1 hadoop hadoop 10692 Jun 18 15:28 spark-3.2.1-bin-hadoop3.2/conf/ozone-site.xml{code}
> However, Spark SQL fails to access Ozone data. The error is as follows:
> {code:java}
> 22/06/19 07:15:59 DEBUG FileSystem: Starting: Acquiring creator semaphore for ofs://cluster1/tgwarehouse
> 22/06/19 07:15:59 DEBUG FileSystem: Acquiring creator semaphore for ofs://cluster1/tgwarehouse: duration 0:00.000s
> 22/06/19 07:15:59 DEBUG Tracer: sampler.classes = ; loaded no samplers
> 22/06/19 07:15:59 DEBUG Tracer: span.receiver.classes = ; loaded no span receivers
> 22/06/19 07:15:59 DEBUG FileSystem: Starting: Creating FS ofs://cluster1/tgwarehouse
> 22/06/19 07:15:59 DEBUG FileSystem: Loading filesystems
> 22/06/19 07:15:59 DEBUG FileSystem: nullscan:// = class org.apache.hadoop.hive.ql.io.NullScanFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hive-exec-2.3.9-core.jar
> 22/06/19 07:15:59 DEBUG FileSystem: file:// = class org.apache.hadoop.fs.LocalFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: file:// = class org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hive-exec-2.3.9-core.jar
> 22/06/19 07:15:59 DEBUG FileSystem: viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: har:// = class org.apache.hadoop.fs.HarFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/hadoop-client-api-3.3.1.jar
> 22/06/19 07:15:59 DEBUG FileSystem: o3fs:// = class org.apache.hadoop.fs.ozone.OzoneFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> 22/06/19 07:15:59 DEBUG FileSystem: ofs:// = class org.apache.hadoop.fs.ozone.RootedOzoneFileSystem from /opt/hadoop/kyuubi/20220618/apache-kyuubi-1.6.0-SNAPSHOT-bin-spark-3.2/externals/spark-3.2.1-bin-hadoop3.2/jars/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
> 22/06/19 07:15:59 DEBUG FileSystem: Looking for FS supporting ofs
> 22/06/19 07:15:59 DEBUG FileSystem: looking for configuration option fs.ofs.impl
> 22/06/19 07:15:59 DEBUG FileSystem: Filesystem ofs defined in configuration option
> 22/06/19 07:15:59 DEBUG FileSystem: FS for ofs is class org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
> 22/06/19 07:15:59 DEBUG Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker@66933239
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache: Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1: org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 1, nodeId: om2
> 22/06/19 07:15:59 DEBUG RetryInvocationHandler: java.lang.ClassCastException: org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message, while invoking $Proxy26.submitRequest over nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862. Trying to failover immediately.
> java.lang.ClassCastException: org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
>         at com.sun.proxy.$Proxy26.submitRequest(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
>         at com.sun.proxy.$Proxy26.submitRequest(Unknown Source)
>         at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
>         at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:282)
>         at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1440)
>         at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:235)
>         at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
>         at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:114)
>         at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:179)
>         at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
>         at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:92)
>         at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:149)
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
>         at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
>         at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
>         at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$.hadoopFSsToAccess(HadoopFSDelegationTokenProvider.scala:176)
>         at org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:50)
>         at org.apache.spark.deploy.security.HadoopDelegationTokenManager.$anonfun$obtainDelegationTokens$2(HadoopDelegationTokenManager.scala:164)
>         at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
>         at scala.collection.Iterator.foreach(Iterator.scala:943)
>         at scala.collection.Iterator.foreach$(Iterator.scala:943)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
>         at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
>         at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
>         at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
>         at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
>         at org.apache.spark.deploy.security.HadoopDelegationTokenManager.org$apache$spark$deploy$security$HadoopDelegationTokenManager$$obtainDelegationTokens(HadoopDelegationTokenManager.scala:162)
>         at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$2.run(HadoopDelegationTokenManager.scala:148)
>         at org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anon$2.run(HadoopDelegationTokenManager.scala:146)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
>         at org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:146)
>         at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$start$1(CoarseGrainedSchedulerBackend.scala:555)
>         at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$start$1$adapted(CoarseGrainedSchedulerBackend.scala:549)
>         at scala.Option.foreach(Option.scala:407)
>         at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.start(CoarseGrainedSchedulerBackend.scala:549)
>         at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.start(KubernetesClusterSchedulerBackend.scala:95)
>         at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
>         at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
>         at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
>         at scala.Option.getOrElse(Option.scala:189)
>         at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
>         at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:186)
>         at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:277)
>         at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>         at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>         at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
>         at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
>         at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
>         at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>         at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>         at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0 to om2:1
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache: Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 2, nodeId: om3
> 22/06/19 07:15:59 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 1 failover attempts. Trying to failover immediately.
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1 to om3:2
> 22/06/19 07:15:59 DEBUG Client: getting client out of cache: Client-3da530302dcb460d8e283786621c481f
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:15:59 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 0, nodeId: om1
> 22/06/19 07:15:59 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om3,nodeAddress=tg-local-bdworker-3.tg.mt.com:9862 after 2 failover attempts. Trying to failover after sleeping for 2000ms.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om3:2 to om1:0
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 1, nodeId: om2
> 22/06/19 07:16:01 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862 after 3 failover attempts. Trying to failover immediately.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0 to om2:1
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 2, nodeId: om3
> 22/06/19 07:16:01 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 4 failover attempts. Trying to failover immediately.
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1 to om3:2
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:16:01 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 0, nodeId: om1
> 22/06/19 07:16:01 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om3,nodeAddress=tg-local-bdworker-3.tg.mt.com:9862 after 5 failover attempts. Trying to failover after sleeping for 2000ms.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om3:2 to om1:0
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om1: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 1, nodeId: om2
> 22/06/19 07:16:03 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om1,nodeAddress=tg-local-bdworker-1.tg.mt.com:9862 after 6 failover attempts. Trying to failover immediately.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om1:0 to om2:1
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om2: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 2, nodeId: om3
> 22/06/19 07:16:03 INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy26.submitRequest over nodeId=om2,nodeAddress=tg-local-bdworker-2.tg.mt.com:9862 after 7 failover attempts. Trying to failover immediately.
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Failing over OM from om2:1 to om3:2
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: RetryProxy: OM om3: null
> 22/06/19 07:16:03 DEBUG OMFailoverProxyProvider: Incrementing OM proxy index to 0, nodeId: om1{code}
> If I use the following method to build Spark, the access is successful:
> {code:java}
> mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=3.2.2 -Phive -Phive-thriftserver -Pkubernetes -DskipTests clean package{code}
>  
> Just use -Phadoop-2.7 to compile and ${SPARK_HOME}/jars contains the hadoop jar:
> {code:java}
> $ ll hadoop-*
> rw-rr- 1 hadoop hadoop   60199 Apr 21 11:15 hadoop-annotations-3.2.2.jar
> rw-rr- 1 hadoop hadoop  138914 Apr 21 11:15 hadoop-auth-3.2.2.jar
> rw-rr- 1 hadoop hadoop   44107 Apr 21 11:15 hadoop-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop 4182881 Apr 21 11:15 hadoop-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop 5139052 Apr 21 11:15 hadoop-hdfs-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop  805828 Apr 21 11:15 hadoop-mapreduce-client-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop 1658798 Apr 21 11:15 hadoop-mapreduce-client-core-3.2.2.jar
> rw-rr- 1 hadoop hadoop   85841 Apr 21 11:15 hadoop-mapreduce-client-jobclient-3.2.2.jar
> rw-rr- 1 hadoop hadoop 3289927 Apr 21 11:15 hadoop-yarn-api-3.2.2.jar
> rw-rr- 1 hadoop hadoop  325110 Apr 21 11:15 hadoop-yarn-client-3.2.2.jar
> rw-rr- 1 hadoop hadoop 2919326 Apr 21 11:15 hadoop-yarn-common-3.2.2.jar
> rw-rr- 1 hadoop hadoop   80795 Apr 21 11:15 hadoop-yarn-server-web-proxy-3.2.2.jar{code}
> Does not currently support Shaded Hadoop jars in Ozone?
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org