hbase 2.1.4: insert error: cannot get replica 0

[Michael <mi...@fantasymail.de>]

Hi,

I try to run an insert job implemented in java, but it doesn't work.

It does run for about 30min, inserting with around 6600 record per
second. After that I get exceptions:
hbase.client.AsyncRequestFutureImpl: "Cannot get replica 0 location for
{....}


After getting many of these the implort stops.

I collect 200 puts, then insert them as a batch in a java thread.

I have an eight node cluster, with five region servers on 8-core
machines with 64GB memory and 7TB hdd.

CPU during insert is quite low.

What could be a reason for that?

Thanks for helping,
 Michael

Re: hbase 2.1.4: insert error: cannot get replica 0

[Biju N <bi...@gmail.com>]

Major compactions can take long time depending on data size. Are you moving
from an earlier version of HBase to 2.1.4 i.e. did this loading process
work fine earlier? When you mentioned "200 puts as a batch in a java
thread" assuming you are using HBase batch mutation? How much memory out of
the 64 GB allocated to HBase? Any frequent long GC pauses (information
should be available in the RS GC logs)? If you are loading a fresh table is
it pre-split to take advantage of all the RSes you have? Any other workload
during this data load?

Thanks

On Fri, Apr 26, 2019 at 8:03 AM Michael <mi...@fantasymail.de> wrote:

> The logs show nothing which looks like an error.
>
> Are long compactions which take up to 100sec ok?
>
> Otherwise I don't find any error or exceptions in the logs.
>
> cheers
>  Michael
>
>
> Am 25.04.19 um 07:10 schrieb Stack:
> > Check the logs on the servers? Anything untoward? For example, if you
> look
> > in the master logs, any complaints about regions having trouble onlining?
> >
>
>
>
> > What version of hbase?
>
> version 2.1.4
>
>

Re: hbase 2.1.4: insert error: cannot get replica 0

[Michael <mi...@fantasymail.de>]

The logs show nothing which looks like an error.

Are long compactions which take up to 100sec ok?

Otherwise I don't find any error or exceptions in the logs.

cheers
 Michael


Am 25.04.19 um 07:10 schrieb Stack:
> Check the logs on the servers? Anything untoward? For example, if you look
> in the master logs, any complaints about regions having trouble onlining?
>



> What version of hbase?

version 2.1.4

Re: hbase 2.1.4: insert error: cannot get replica 0

[Stack <st...@duboce.net>]

Check the logs on the servers? Anything untoward? For example, if you look
in the master logs, any complaints about regions having trouble onlining?

What version of hbase?

Thanks,
S

On Wed, Apr 24, 2019 at 12:31 AM Michael <mi...@fantasymail.de> wrote:

> Hi,
>
> I try to run an insert job implemented in java, but it doesn't work.
>
> It does run for about 30min, inserting with around 6600 record per
> second. After that I get exceptions:
> hbase.client.AsyncRequestFutureImpl: "Cannot get replica 0 location for
> {....}
>
>
> After getting many of these the implort stops.
>
> I collect 200 puts, then insert them as a batch in a java thread.
>
> I have an eight node cluster, with five region servers on 8-core
> machines with 64GB memory and 7TB hdd.
>
> CPU during insert is quite low.
>
> What could be a reason for that?
>
> Thanks for helping,
>  Michael
>
>