You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Miklos Gergely (Jira)" <ji...@apache.org> on 2020/10/01 15:35:00 UTC
[jira] [Comment Edited] (HBASE-25140) HBase test mini cluster is
working only with Hadoop 2.8.0 - 3.0.3
[ https://issues.apache.org/jira/browse/HBASE-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205614#comment-17205614 ]
Miklos Gergely edited comment on HBASE-25140 at 10/1/20, 3:34 PM:
------------------------------------------------------------------
Yes, I meant hadoop 2.4.1, but I as it is not supported, it is not relelvant. At Flink our nightly tests run with hadoop 2.4.1 and 3.1.3 and they both fail, for 3.1.3 check this: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7093&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51]
This is the maven output, as you can see it dislays onlly that "Master not initialized after 200000ms". You can find the actual logs with the min cluster failure here at the attached log file, here is the error:
{code:java}
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:496)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:116)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:576)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:571)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:584)
at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:643)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:767)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:501)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:442)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:156)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:61)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:284)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2181)
at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:133)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
One way to reproduce this:
# git clone [https://github.com/apache/flink.git]
# mvn clean install -DskipTests -Dfast # this will take about 30 minutes
# edit flink-connectors/flink-connector-hbase-2.2/src/test/java/org/apache/flink/connector/hbase2/util/HBaseTestingClusterAutoStarter.java - modify the range at line 57 to include 3.1.3
# cd flink-connectors/flink-connector-hbase-2.2
# mvn verify -Dit.test=HBaseConnectorITCase -Dhadoop.version=3.1.3
was (Author: mgergely):
Yes, I meant hadoop 2.4.1, but I as it is not supported, it is not relelvant. At Flink our nightly tests run with hadoop 2.4.1 and 3.1.3 and they both fail, for 3.1.3 check this: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7093&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51]
This is the maven output, as you can see it dislays onlly that "Master not initialized after 200000ms". You can find the actual logs with the min cluster failure here at the attached log file, here is the error:
{code:java}
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:496)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:116)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:576)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:571)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:584)
at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:643)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:767)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:501)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:442)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:156)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:61)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:284)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2181)
at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:133)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
One way to reproduce this:
> HBase test mini cluster is working only with Hadoop 2.8.0 - 3.0.3
> -----------------------------------------------------------------
>
> Key: HBASE-25140
> URL: https://issues.apache.org/jira/browse/HBASE-25140
> Project: HBase
> Issue Type: Bug
> Components: documentation, hadoop2, test
> Affects Versions: 2.2.3
> Reporter: Miklos Gergely
> Priority: Major
>
> Running HBaseTestingUtility.startMiniCluster() on HBase 2.2.3 works only with hadoop version range 2.8.0 - 3.0.3, for example with 2.4.1 the following exception occurs:
>
> {code:java}
> 21:49:04,124 [RS:0;71af2d647bb3:35715] ERROR org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper [] - Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.21:49:04,124 [RS:0;71af2d647bb3:35715] ERROR org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper [] - Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.beginFileLease(long, org.apache.hadoop.hdfs.DFSOutputStream) at java.lang.Class.getDeclaredMethod(Class.java:2130) ~[?:1.8.0_242] at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createLeaseManager(FanOutOneBlockAsyncDFSOutputHelper.java:198) ~[hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.<clinit>(FanOutOneBlockAsyncDFSOutputHelper.java:274) [hbase-server-2.2.3.jar:2.2.3] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_242] at java.lang.Class.forName(Class.java:264) [?:1.8.0_242] at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:136) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:198) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1871) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1589) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:157) [hbase-server-2.2.3-tests.jar:2.2.3] at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001) [hbase-server-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) [hbase-server-2.2.3-tests.jar:2.2.3] at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) [hbase-server-2.2.3-tests.jar:2.2.3] at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) [hbase-server-2.2.3-tests.jar:2.2.3] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_242] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1536) [hadoop-common-2.4.1.jar:?] at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) [hbase-common-2.2.3.jar:2.2.3] at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) [hbase-server-2.2.3-tests.jar:2.2.3] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
> {code}
> Also upon failure during maven run it would be great if the actual exception would be displayed, not just that "Master not initialized after 200000ms".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)