You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Samudrala, Ranganath [USA] via user" <us...@accumulo.apache.org> on 2023/02/21 18:43:26 UTC

Problem creating tables in Accumulo

Hello,

Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS and table data is stored in Amazon S3. No attempt has been made yet to store/insert table data. But, when I attempt to create a table in Accumulo via Java API, I see the following error.



WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc
umulo-k8s.svc.cluster.local:9999 (0) for at least 120033 ms


I see the below error in T-SERVER logs:

2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet +r<< hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables/+r/root_tablet
java.lang.NullPointerException
        at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60)
        at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:341)
        at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2504)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:750)
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< reporting failure to master
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in 600.00 seconds
2023-02-21 18:07:03,210 WARN [SimpleTimer-0] [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater than ten (30) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement.


The installed configuration is made up of 2 instances of HDFS data nodes, 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo master.

> config -t accumulo.root
.
default    | table.file.replication ........................ | 0
site       |    @override .................................. | 2
table      |    @override .................................. | 5
.
.
> config -t accumulo.metadata
.
default    | table.file.replication ........................ | 0
site       |    @override .................................. | 2
table      |    @override .................................. | 5
.

How do I fix this problem?

Thanks in advance.
Ranga

Re: [External] RE: Problem creating tables in Accumulo

Posted by "Samudrala, Ranganath [USA] via user" <us...@accumulo.apache.org>.
Thank you every one for the response.

I think the problem is related to slow / intermittent response from Amazon S3 due the network situation we are in. I am trying to use an internal S3 bucket to see if the problem persists.

thanks again.
Ranga
________________________________
From: dlmarion@comcast.net <dl...@comcast.net>
Sent: Wednesday, February 22, 2023 2:59 PM
To: user@accumulo.apache.org <us...@accumulo.apache.org>; Samudrala, Ranganath [USA] <Sa...@bah.com>
Subject: [External] RE: Problem creating tables in Accumulo


It looks like the issue is that `maxTime` is not being set at [0], so `time` is not being set at [1]. If I had to guess, Accumulo is having an issue talking with ZooKeeper. Were there errors in the log when initializing Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods and confirm connectivity with Accumulo?

[0] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java#L159<https://urldefense.com/v3/__https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java*L159__;Iw!!May37g!LFm9Bahei8b-iJypacQhWPQOTL5v3dvrCk-sRw87xEjf0D4tmTTJ81ySH0JaG_Md_oqrAVfSfFbDa0hTSd435920$>
[1] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java#L166<https://urldefense.com/v3/__https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java*L166__;Iw!!May37g!LFm9Bahei8b-iJypacQhWPQOTL5v3dvrCk-sRw87xEjf0D4tmTTJ81ySH0JaG_Md_oqrAVfSfFbDa0hTSdE9N0sW$>



From: Samudrala, Ranganath [USA] via user <us...@accumulo.apache.org>
Sent: Tuesday, February 21, 2023 1:43 PM
To: Samudrala, Ranganath [USA] via user <us...@accumulo.apache.org>
Subject: Problem creating tables in Accumulo



Hello,



Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS and table data is stored in Amazon S3. No attempt has been made yet to store/insert table data. But, when I attempt to create a table in Accumulo via Java API, I see the following error.





WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc

umulo-k8s.svc.cluster.local:9999 (0) for at least 120033 ms



I see the below error in T-SERVER logs:

2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet +r<< hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables/+r/root_tablet

java.lang.NullPointerException

        at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60)

        at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:341)

        at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2504)

        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)

        at java.lang.Thread.run(Thread.java:750)

2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< reporting failure to master

2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in 600.00 seconds

2023-02-21 18:07:03,210 WARN [SimpleTimer-0] [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater than ten (30) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement.





The installed configuration is made up of 2 instances of HDFS data nodes, 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo master.



> config -t accumulo.root

.

default    | table.file.replication ........................ | 0

site       |    @override .................................. | 2

table      |    @override .................................. | 5

.

.

> config -t accumulo.metadata

.

default    | table.file.replication ........................ | 0

site       |    @override .................................. | 2

table      |    @override .................................. | 5

.



How do I fix this problem?

Thanks in advance.

Ranga

RE: Problem creating tables in Accumulo

Posted by dl...@comcast.net.
It looks like the issue is that `maxTime` is not being set at [0], so `time`
is not being set at [1]. If I had to guess, Accumulo is having an issue
talking with ZooKeeper. Were there errors in the log when initializing
Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods
and confirm connectivity with Accumulo?

[0]
https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or
g/apache/accumulo/tserver/tablet/TabletData.java#L159
[1]
https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or
g/apache/accumulo/tserver/tablet/TabletData.java#L166

 

From: Samudrala, Ranganath [USA] via user <us...@accumulo.apache.org> 
Sent: Tuesday, February 21, 2023 1:43 PM
To: Samudrala, Ranganath [USA] via user <us...@accumulo.apache.org>
Subject: Problem creating tables in Accumulo

 

Hello,

 

Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS
and table data is stored in Amazon S3. No attempt has been made yet to
store/insert table data. But, when I attempt to create a table in Accumulo
via Java API, I see the following error.

 

 

WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread
"main" stuck on IO to accumulo-masters-0.accumulo-masters.acc
umulo-k8s.svc.cluster.local:9999 (0) for at least 120033 ms

 

I see the below error in T-SERVER logs:

2023-02-21 18:04:09,373 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: exception trying to assign
tablet +r<<
hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables
/+r/root_tablet
java.lang.NullPointerException
        at
org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60
)
        at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:341)
        at
org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.
java:2504)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:750)
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<<
reporting failure to master
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in
600.00 seconds
2023-02-21 18:07:03,210 WARN [SimpleTimer-0]
[org.apache.accumulo.server.ServerUtil]: System swappiness setting is
greater than ten (30) which can cause time-sensitive operations to be
delayed. Accumulo is time sensitive because it needs to maintain distributed
lock agreement.

 

 

The installed configuration is made up of 2 instances of HDFS data nodes, 2
instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo
master.

 

> config -t accumulo.root

.

default    | table.file.replication ........................ | 0 

site       |    @override .................................. | 2

table      |    @override .................................. | 5

.

.

> config -t accumulo.metadata

.

default    | table.file.replication ........................ | 0 

site       |    @override .................................. | 2

table      |    @override .................................. | 5

.

 

How do I fix this problem?


Thanks in advance.

Ranga

Re: Problem creating tables in Accumulo

Posted by Daniel Roberts <dd...@gmail.com>.
Ranga,

Your current error message seems to point to your root tablet not being
able to be created (or assigned).
I would start by trying to verify that HDFS is functional and that your
`instance.volumes` entries include a valid HDFS volume entry.

Does your monitor page report additional errors?

- Daniel

On Tue, Feb 21, 2023 at 1:43 PM Samudrala, Ranganath [USA] via user <
user@accumulo.apache.org> wrote:

> Hello,
>
> Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS
> and table data is stored in Amazon S3. No attempt has been made yet to
> store/insert table data. But, when I attempt to create a table in
> Accumulo via Java API, I see the following error.
>
>
> WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc
> umulo-k8s.svc.cluster.local:9999 (0) for at least 120033 ms
>
>
>
> I see the below error in T-SERVER logs:
>
> 2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet +r<< hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables/+r/root_tablet
> java.lang.NullPointerException
>         at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60)
>         at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:341)
>         at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2504)
>         at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:750)
> 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< reporting failure to master
> 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in 600.00 seconds
> 2023-02-21 18:07:03,210 WARN [SimpleTimer-0] [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater than ten (30) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement.
>
>
>
> The installed configuration is made up of 2 instances of HDFS data nodes,
> 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo
> master.
>
> > config -t accumulo.root
> .
> default    | table.file.replication ........................ | 0
> site       |    @override .................................. | 2
> table      |    @override .................................. | 5
> .
> .
> > config -t accumulo.metadata
> .
> default    | table.file.replication ........................ | 0
> site       |    @override .................................. | 2
> table      |    @override .................................. | 5
> .
>
> How do I fix this problem?
>
> Thanks in advance.
> Ranga
>
>