You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Zhuqi Jin (Jira)" <ji...@apache.org> on 2020/05/26 03:00:01 UTC
[jira] [Updated] (HBASE-24430) upgrading standalone-mode hbase from
2.2 to 2.3 fails because of local filesystem not supporting hflush
[ https://issues.apache.org/jira/browse/HBASE-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhuqi Jin updated HBASE-24430:
------------------------------
Component/s: (was: wal)
Affects Version/s: 3.0.0-alpha-1
Description:
When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to start.
The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,130 INFO [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26 02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more2020-05-26 02:46:33,135 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting shutdown. *****java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more
{code}
It can be reproduced through the following steps:
# Start a single-node Hbase branch-2.2.
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
3. Upgrade the node to branch-2.3/master. After upgraded, as the log suggested, hbase failed to start.
was:
When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to start.
The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,130 INFO [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26 02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more2020-05-26 02:46:33,135 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting shutdown. *****java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more
{code}
It can be reproduced through the following steps:
# Start a single-node Hbase branch-2.2.
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
# Upgrade the node to branch-2.3/master. After upgraded, as the log suggested, hbase failed to start.
> upgrading standalone-mode hbase from 2.2 to 2.3 fails because of local filesystem not supporting hflush
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-24430
> URL: https://issues.apache.org/jira/browse/HBASE-24430
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha-1, 2.3.0
> Reporter: Zhuqi Jin
> Priority: Major
>
> When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to start.
> The error message is shown below:
> {code:java}
> 2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,130 INFO [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26 02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more2020-05-26 02:46:33,135 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting shutdown. *****java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more
> {code}
> It can be reproduced through the following steps:
> # Start a single-node Hbase branch-2.2.
> # Use hbase pe to write data.
> {code:java}
> /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
> 3. Upgrade the node to branch-2.3/master. After upgraded, as the log suggested, hbase failed to start.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)