[jira] [Resolved] (HBASE-23861) Reconcile Hadoop version

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-23861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei-Chiu Chuang resolved HBASE-23861.
-------------------------------------
    Resolution: Fixed

Resolve this and leave the fix in master and branch-2.3.

I'll file a Jira to update the user guide when applicable.

 

Thanks!

> Reconcile Hadoop version
> ------------------------
>
>                 Key: HBASE-23861
>                 URL: https://issues.apache.org/jira/browse/HBASE-23861
>             Project: HBase
>          Issue Type: Bug
>          Components: dependencies
>    Affects Versions: 3.0.0
>         Environment: Apache Maven 3.6.1
> Hadoop 3.2.0 and above.
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>            Priority: Major
>             Fix For: 3.0.0, 2.3.0
>
>
> I followed the HBase book http://hbase.apache.org/book.html#maven.build.hadoop and wanted to build HBase (master) on top of Hadoop 3.2/3.3 but tests failed right away.
> Build: 
> {code}
> mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests
> {code}
> Test:
> {code}
> mvn test -Dtest=TestHelloHBase -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1
> {code}
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.296 s <<< FAILURE! - in org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase
> [ERROR] org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase  Time elapsed: 1.284 s  <<< ERROR!
> java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/HdfsConstants$StoragePolicySatisfierMode
> at org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocol.HdfsConstants$StoragePolicySatisfierMode
> at org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)
> {noformat}
> Adding mvn -X parameter, I was able to tell that it was because the hbase-server module includes hadoop-distcp and hadoop-dfs-client 3.1.2 (the default Hadoop 3 dependency version) while it uses version 3.2.1 of other hadoop jars . The classpath conflict (the storage policy satisfier is a new feature in Hadoop 3.2) failed the test.
> This is reproducible on any Hadoop version 3.2 and above. It looks to me the version of hadoop-distcp and hadoop-hdfs-client should be specified at the top level pom (they are specified in hbase-server/pom.xml).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)