You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Ligade, Shailesh [USA]" <Li...@bah.com> on 2022/01/04 16:27:22 UTC
replication table offline issue
Hello,
I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all
the error is:
-------------------------------------
WARN Failed to write work mutations for replication, will retry
clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6
at xxxclient.impl.TabletServerBatchWriter.checkForFailures
....
caused by TableOfflineException: Table accumulo.replication (+rep) is offline
---------------------------
There are no constraints that I am using on any table.
I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before)
if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error
Any suggestion on how to fix this?
Thanks
-S
Re: replication table offline issue
Posted by Christopher <ct...@apache.org>.
This might be a transient issue. It looks like a warning, not an
error. Maybe the table state in ZK is not propagating to all tservers
at the same time? Restarting the cluster could help. You probably
didn't need to add WRITE permission to the table. If I remember
correctly, replication should use the system credentials, not the root
user's, so granting WRITE to the root user shouldn't matter.
On Tue, Jan 4, 2022 at 12:24 PM Ligade, Shailesh [USA]
<Li...@bah.com> wrote:
>
> Sorry this is for accumulo 1.10.0
>
> I am wondering is there a way to delete and recreate the accumulo.replication table. I know it is bit special table..so
>
> Also, will restarting entire cluster solve this? or may be just restarting accumulo master may be?
>
> Since rolling restart of tservers is bit lengthy process for us just wanted to make sure it may resolve it or not..
>
> -S
> ________________________________
> From: Ligade, Shailesh [USA]
> Sent: Tuesday, January 4, 2022 11:27 AM
> To: user@accumulo.apache.org <us...@accumulo.apache.org>
> Subject: replication table offline issue
>
> Hello,
>
> I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all
>
> the error is:
> -------------------------------------
>
> WARN Failed to write work mutations for replication, will retry
> clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6
> at xxxclient.impl.TabletServerBatchWriter.checkForFailures
> ....
> caused by TableOfflineException: Table accumulo.replication (+rep) is offline
> ---------------------------
>
> There are no constraints that I am using on any table.
> I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before)
> if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error
>
> Any suggestion on how to fix this?
>
> Thanks
>
> -S
>
>
Re: replication table offline issue
Posted by "Ligade, Shailesh [USA]" <Li...@bah.com>.
Thanks Ed/Chris,
yes replication is table is hosted only one tablet server and I looked at that tablet server, there are no errors.
I am seeing all this errors in active master server log.
zkcli to state the replication table showed its online.
I guess I will restart masters, if that doesn't work then restart that just one tserver holding the replication tablet and if that fails then restart entire cluster 🙁
Will remove write grant from root and replication user.
Appreciated your help
-S
________________________________
From: dev1 <de...@etcoleman.com>
Sent: Tuesday, January 4, 2022 1:18 PM
To: 'user@accumulo.apache.org' <us...@accumulo.apache.org>
Subject: [External] RE: replication table offline issue
Deleting / recreating the replication table should not be necessary and in any case you very likely cannot delete / create the accumulo.replication table – the shell will error on the delete because it is in the accumulo namespace.
Is the replication table hosted on a single tserver? Are there any exceptions in the log for that server? (or any of the tservers host it if hosted across multiple tservers)
Have you restarted the client? It looks like the exception fragment has client in the classname. What log is that exception occurring?
You can try restarting the master(s)
The monitor shows the replication table is online? Can you check in ZooKeeper (using the zkCli.sh)
* get /accumulo/[instance id]/tables/+rep/state
That should return the text ONLINE
If the replication table is on a single tserver, then you might be able to just restart that server rather than needing to do a rolling restart of the cluster. If there a no errors in the tserver log this seems unlikely to help.
Ed Coleman
From: Ligade, Shailesh [USA] <Li...@bah.com>
Sent: Tuesday, January 4, 2022 12:24 PM
To: user@accumulo.apache.org
Subject: Re: replication table offline issue
Sorry this is for accumulo 1.10.0
I am wondering is there a way to delete and recreate the accumulo.replication table. I know it is bit special table..so
Also, will restarting entire cluster solve this? or may be just restarting accumulo master may be?
Since rolling restart of tservers is bit lengthy process for us just wanted to make sure it may resolve it or not..
-S
________________________________
From: Ligade, Shailesh [USA]
Sent: Tuesday, January 4, 2022 11:27 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org> <us...@accumulo.apache.org>>
Subject: replication table offline issue
Hello,
I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all
the error is:
-------------------------------------
WARN Failed to write work mutations for replication, will retry
clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6
at xxxclient.impl.TabletServerBatchWriter.checkForFailures
....
caused by TableOfflineException: Table accumulo.replication (+rep) is offline
---------------------------
There are no constraints that I am using on any table.
I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before)
if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error
Any suggestion on how to fix this?
Thanks
-S
RE: replication table offline issue
Posted by dev1 <de...@etcoleman.com>.
Deleting / recreating the replication table should not be necessary and in any case you very likely cannot delete / create the accumulo.replication table - the shell will error on the delete because it is in the accumulo namespace.
Is the replication table hosted on a single tserver? Are there any exceptions in the log for that server? (or any of the tservers host it if hosted across multiple tservers)
Have you restarted the client? It looks like the exception fragment has client in the classname. What log is that exception occurring?
You can try restarting the master(s)
The monitor shows the replication table is online? Can you check in ZooKeeper (using the zkCli.sh)
* get /accumulo/[instance id]/tables/+rep/state
That should return the text ONLINE
If the replication table is on a single tserver, then you might be able to just restart that server rather than needing to do a rolling restart of the cluster. If there a no errors in the tserver log this seems unlikely to help.
Ed Coleman
From: Ligade, Shailesh [USA] <Li...@bah.com>
Sent: Tuesday, January 4, 2022 12:24 PM
To: user@accumulo.apache.org
Subject: Re: replication table offline issue
Sorry this is for accumulo 1.10.0
I am wondering is there a way to delete and recreate the accumulo.replication table. I know it is bit special table..so
Also, will restarting entire cluster solve this? or may be just restarting accumulo master may be?
Since rolling restart of tservers is bit lengthy process for us just wanted to make sure it may resolve it or not..
-S
________________________________
From: Ligade, Shailesh [USA]
Sent: Tuesday, January 4, 2022 11:27 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org> <us...@accumulo.apache.org>>
Subject: replication table offline issue
Hello,
I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all
the error is:
-------------------------------------
WARN Failed to write work mutations for replication, will retry
clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6
at xxxclient.impl.TabletServerBatchWriter.checkForFailures
....
caused by TableOfflineException: Table accumulo.replication (+rep) is offline
---------------------------
There are no constraints that I am using on any table.
I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before)
if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error
Any suggestion on how to fix this?
Thanks
-S
Re: replication table offline issue
Posted by "Ligade, Shailesh [USA]" <Li...@bah.com>.
Sorry this is for accumulo 1.10.0
I am wondering is there a way to delete and recreate the accumulo.replication table. I know it is bit special table..so
Also, will restarting entire cluster solve this? or may be just restarting accumulo master may be?
Since rolling restart of tservers is bit lengthy process for us just wanted to make sure it may resolve it or not..
-S
________________________________
From: Ligade, Shailesh [USA]
Sent: Tuesday, January 4, 2022 11:27 AM
To: user@accumulo.apache.org <us...@accumulo.apache.org>
Subject: replication table offline issue
Hello,
I setup replication and ran 'online accumulo.replication' however i n master i keep on getting error stating accumulo.replication is offline. I can scan accumulo.replication table it has no data at all
the error is:
-------------------------------------
WARN Failed to write work mutations for replication, will retry
clinet.MutationRejectedException: # constraint violations: 0 security codes: {} # server errors 0 # exceptions 6
at xxxclient.impl.TabletServerBatchWriter.checkForFailures
....
caused by TableOfflineException: Table accumulo.replication (+rep) is offline
---------------------------
There are no constraints that I am using on any table.
I added grants for root as well as my replication user for accumulo.replication Tbale.WRITE (there was only Table.READ before)
if i run offline accumulo.replication i can see it is offline and then i can bring in online again however i still keep getting error
Any suggestion on how to fix this?
Thanks
-S