You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kal Stevens <ka...@gmail.com> on 2024/11/15 01:18:50 UTC
Problem starting up hbase
I am not sure what is wrong with my hbase setup. And I am not sure where
to look to investigate this.
I do not see anything wrong in the logs, so I am not sure where to go from
here.
I was having a problem with zookeeper, but I may have gotten around that, I
am not 100% sure.
I am getting this error message on the hbasemaster
2024-11-14T17:08:06,945 WARN [master/dn2:16000:becomeActiveMaster]
master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740
state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197};
ServerCrashProcedures=true. Master startup cannot progress, in
holding-pattern until region onlined.
2024-11-14T17:09:06,946 WARN [master/dn2:16000:becomeActiveMaster]
master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740
state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197};
ServerCrashProcedures=true. Master startup cannot progress, in
holding-pattern until region onlined.
But my region servers are started. And when I restarted the hbasemaster, I
got a connection refused on the region server.
Here is a snippet from the regionserver logs
2024-11-14T16:53:20,274 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=CompactionThroughputTuner, period=60000,
unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,275 INFO [regionserver/dn2:16020]
regionserver.HRegionServer: CompactionChecker runs every PT10S
2024-11-14T16:53:20,298 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=CompactedHFilesCleaner, period=120000,
unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=CompactionChecker, period=10000,
unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=MemstoreFlusherChore, period=10000,
unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is
enabled.
2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=BrokenStoreFileCleaner, period=21600000,
unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=dn2,16020,1731631998737-MobFileCleanerChore,
period=86400, unit=SECONDS is enabled.
2024-11-14T16:53:20,318 INFO [regionserver/dn2:16020]
regionserver.HeapMemoryManager: Starting, tuneOn=false
2024-11-14T16:53:20,320 INFO [regionserver/dn2:16020] hbase.ChoreService:
Chore ScheduledChore name=dn2,16020,1731631998737-HeapMemoryTunerChore,
period=60000, unit=MILLISECONDS is enabled.
2024-11-14T16:53:20,321 INFO [regionserver/dn2:16020]
regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk
size 2 MB, max count 2890, initial count 0
2024-11-14T16:53:20,322 INFO [regionserver/dn2:16020]
regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk
size 204.80 KB, max count 3212, initial count 0
2024-11-14T16:53:20,334 INFO [regionserver/dn2:16020]
regionserver.Replication: dn2,16020,1731631998737 started
2024-11-14T16:53:20,335 INFO [regionserver/dn2:16020]
regionserver.HRegionServer: Serving as dn2,16020,1731631998737, RpcServer
on dn2/192.168.1.109:16020, sessionid=0x3000036514f0002
2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020]
quotas.RegionServerRpcQuotaManager: Quota support disabled
2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020]
quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting
space quota manager.
2024-11-14T16:53:23,377 INFO [regionserver/dn2:16020] wal.AbstractFSWAL:
WAL configuration: blocksize=256 MB, rollsize=128 MB,
prefix=dn2%2C16020%2C1731631998737, suffix=,
logDir=hdfs://dn1/hbase/WALs/dn2,16020,1731631998737,
archiveDir=hdfs://dn1/hbase/oldWALs, maxLogs=100
2024-11-14T16:53:23,383 INFO [regionserver/dn2:16020]
monitor.StreamSlowMonitor: New stream slow monitor
dn2%2C16020%2C1731631998737.1731632003381
2024-11-14T16:53:23,445 INFO [regionserver/dn2:16020] wal.AbstractFSWAL:
New WAL
/hbase/WALs/dn2,16020,1731631998737/dn2%2C16020%2C1731631998737.1731632003381
It is creating the directory in hdfs
*kal@dn1*:*/hadoop/deploy/hadoop/bin*$ ./hdfs dfs -ls /hbase/WALs
Found 846 items
drwxr-xr-x - root supergroup 0 2024-11-13 18:04
/hbase/WALs/dn1,16020,1731549858040
drwxr-xr-x - root supergroup 0 2024-11-13 22:08
/hbase/WALs/dn1,16020,1731564499960
drwxr-xr-x - root supergroup 0 2024-11-13 23:05
/hbase/WALs/dn1,16020,1731567827103
drwxr-xr-x - root supergroup 0 2024-11-13 23:08
/hbase/WALs/dn1,16020,1731567978031
drwxr-xr-x - root supergroup 0 2024-11-13 23:10
/hbase/WALs/dn1,16020,1731568139080
drwxr-xr-x - root supergroup 0 2024-11-13 23:13
/hbase/WALs/dn1,16020,1731568290018
This is my hbase-site.xml
<configuration>
<property>
<name>hbase.cluster.distributed</name>
<value>false</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>./tmp</value>
</property>
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://dn1/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>dn1,dn2,dn3</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/hadoop/data/hbase/zookeeper</value>
</property>
</configuration>
And my regionservers
dn1
dn2
dn3
Re: Problem starting up hbase
Posted by Dávid Paksy <pa...@gmail.com>.
Hi Kal,
I'm not an HBase expert by any means, just I spotted that you have the
"hbase.cluster.distributed" property twice in your config: one with false
and then one with true.
I assume that you want to have a fully distributed mode, in this case
you'll need hbase.cluster.distributed=true.
Maybe this is not problem. Hope this helps though.
Best Regards,
Dávid
Kal Stevens <ka...@gmail.com> ezt írta (időpont: 2024. nov. 15., P,
2:20):
> I am not sure what is wrong with my hbase setup. And I am not sure where
> to look to investigate this.
> I do not see anything wrong in the logs, so I am not sure where to go from
> here.
> I was having a problem with zookeeper, but I may have gotten around that, I
> am not 100% sure.
>
> I am getting this error message on the hbasemaster
>
> 2024-11-14T17:08:06,945 WARN [master/dn2:16000:becomeActiveMaster]
> master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740
> state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.
>
> 2024-11-14T17:09:06,946 WARN [master/dn2:16000:becomeActiveMaster]
> master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740
> state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.
>
> But my region servers are started. And when I restarted the hbasemaster, I
> got a connection refused on the region server.
> Here is a snippet from the regionserver logs
>
> 2024-11-14T16:53:20,274 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=CompactionThroughputTuner, period=60000,
> unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,275 INFO [regionserver/dn2:16020]
> regionserver.HRegionServer: CompactionChecker runs every PT10S
>
> 2024-11-14T16:53:20,298 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=CompactedHFilesCleaner, period=120000,
> unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=CompactionChecker, period=10000,
> unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=MemstoreFlusherChore, period=10000,
> unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is
> enabled.
>
> 2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=BrokenStoreFileCleaner, period=21600000,
> unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=dn2,16020,1731631998737-MobFileCleanerChore,
> period=86400, unit=SECONDS is enabled.
>
> 2024-11-14T16:53:20,318 INFO [regionserver/dn2:16020]
> regionserver.HeapMemoryManager: Starting, tuneOn=false
>
> 2024-11-14T16:53:20,320 INFO [regionserver/dn2:16020] hbase.ChoreService:
> Chore ScheduledChore name=dn2,16020,1731631998737-HeapMemoryTunerChore,
> period=60000, unit=MILLISECONDS is enabled.
>
> 2024-11-14T16:53:20,321 INFO [regionserver/dn2:16020]
> regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk
> size 2 MB, max count 2890, initial count 0
>
> 2024-11-14T16:53:20,322 INFO [regionserver/dn2:16020]
> regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk
> size 204.80 KB, max count 3212, initial count 0
>
> 2024-11-14T16:53:20,334 INFO [regionserver/dn2:16020]
> regionserver.Replication: dn2,16020,1731631998737 started
>
> 2024-11-14T16:53:20,335 INFO [regionserver/dn2:16020]
> regionserver.HRegionServer: Serving as dn2,16020,1731631998737, RpcServer
> on dn2/192.168.1.109:16020, sessionid=0x3000036514f0002
>
> 2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020]
> quotas.RegionServerRpcQuotaManager: Quota support disabled
>
> 2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020]
> quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting
> space quota manager.
>
> 2024-11-14T16:53:23,377 INFO [regionserver/dn2:16020] wal.AbstractFSWAL:
> WAL configuration: blocksize=256 MB, rollsize=128 MB,
> prefix=dn2%2C16020%2C1731631998737, suffix=,
> logDir=hdfs://dn1/hbase/WALs/dn2,16020,1731631998737,
> archiveDir=hdfs://dn1/hbase/oldWALs, maxLogs=100
>
> 2024-11-14T16:53:23,383 INFO [regionserver/dn2:16020]
> monitor.StreamSlowMonitor: New stream slow monitor
> dn2%2C16020%2C1731631998737.1731632003381
>
> 2024-11-14T16:53:23,445 INFO [regionserver/dn2:16020] wal.AbstractFSWAL:
> New WAL
>
> /hbase/WALs/dn2,16020,1731631998737/dn2%2C16020%2C1731631998737.1731632003381
>
>
>
>
> It is creating the directory in hdfs
>
> *kal@dn1*:*/hadoop/deploy/hadoop/bin*$ ./hdfs dfs -ls /hbase/WALs
>
> Found 846 items
>
> drwxr-xr-x - root supergroup 0 2024-11-13 18:04
> /hbase/WALs/dn1,16020,1731549858040
>
> drwxr-xr-x - root supergroup 0 2024-11-13 22:08
> /hbase/WALs/dn1,16020,1731564499960
>
> drwxr-xr-x - root supergroup 0 2024-11-13 23:05
> /hbase/WALs/dn1,16020,1731567827103
>
> drwxr-xr-x - root supergroup 0 2024-11-13 23:08
> /hbase/WALs/dn1,16020,1731567978031
>
> drwxr-xr-x - root supergroup 0 2024-11-13 23:10
> /hbase/WALs/dn1,16020,1731568139080
>
> drwxr-xr-x - root supergroup 0 2024-11-13 23:13
> /hbase/WALs/dn1,16020,1731568290018
>
>
>
>
> This is my hbase-site.xml
>
> <configuration>
>
> <property>
>
> <name>hbase.cluster.distributed</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>hbase.tmp.dir</name>
>
> <value>./tmp</value>
>
> </property>
>
> <property>
>
> <name>hbase.unsafe.stream.capability.enforce</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>hbase.rootdir</name>
>
> <value>hdfs://dn1/hbase</value>
>
> </property>
>
> <property>
>
> <name>hbase.cluster.distributed</name>
>
> <value>true</value>
>
> </property>
>
> <property>
>
> <name>hbase.zookeeper.quorum</name>
>
> <value>dn1,dn2,dn3</value>
>
> </property>
>
> <property>
>
> <name>hbase.zookeeper.property.dataDir</name>
>
> <value>/hadoop/data/hbase/zookeeper</value>
>
> </property>
>
>
> </configuration>
>
>
>
> And my regionservers
>
> dn1
>
> dn2
>
> dn3
>