You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Lee King <yu...@gmail.com> on 2017/09/06 06:04:27 UTC
Service unavailable: Transaction failed, tablet 2758e5c68e974b92a3060db8575f3621
transaction memory consumption (67031036) has exceeded its limit (67108864)
or the limit of an ancestral tracker
We got an error about :Service unavailable: Transaction failed, tablet
2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036)
has exceeded its limit (67108864) or the limit of an ancestral tracker.It
looks like https://issues.apache.org/jira/browse/KUDU-1912. and the bug
will be fix at 1.5,but out version is 1.4,Is there any affect for kudu
stablity or data consistency?
Re: Service unavailable: Transaction failed, tablet
2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036)
has exceeded its limit (67108864) or the limit of an ancestral tracker
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Lee,
It seems this email never got a response, sorry about that. Did you figure
out a solution that you can share with the group?
Thanks
-Todd
On Wed, Sep 6, 2017 at 6:11 AM, Lee King <yu...@gmail.com> wrote:
> I use kudu command line tool cluster ksck, the error detail:
> Fetched info from all 10 Tablet Servers
> Tablet 2758e5c68e974b92a3060db8575f3621 of table 'hwx_log' is
> under-replicated: 1 replica(s) not RUNNING
> 067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): bad state
> State: FAILED
> Data state: TABLET_DATA_READY
> Last status: Service unavailable: Transaction failed, tablet
> 2758e5c68e974b92a3060db8575f3621 transaction memory consumption
> (67031036) has exceeded its limit (67108864) or the limit of an ancestral
> tracker
> ad1ea284caff4b07a705c9156b0811cd (cloud-ocean-kudu-01:7050): RUNNING
> c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING
> [LEADER]
>
> 3 replicas' active configs differ from the master's.
> All the peers reported by the master and tablet servers are:
> A = 067e1e7245154f0fb2720dec6c77feec
> B = ad1ea284caff4b07a705c9156b0811cd
> C = c4ed5cb73f5644a8804d3abc976d02f8
>
> The consensus matrix is:
> Config source | Voters | Current term | Config index | Committed?
> ---------------+--------------+--------------+--------------+------------
> master | A B C* | | | Yes
> A | A B C | 10 | 13049 | Yes
> B | A B C | 10 | 13049 | Yes
> C | A C | 10 | 13692 | No
> Tablet 5f996309d53445769c7db89abaa05845 of table 'hwx_log' is
> under-replicated: 1 replica(s) not RUNNING
> 067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): RUNNING
> [LEADER]
> 43d450fd58794f008caca58d44a35ff8 (cloud-ocean-kudu-03:7050): bad state
> State: FAILED
> Data state: TABLET_DATA_READY
> Last status: Service unavailable: Transaction failed, tablet
> 5f996309d53445769c7db89abaa05845 transaction memory consumption
> (66844571) has exceeded its limit (67108864) or the limit of an ancestral
> tracker
> c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING
>
> 1 replicas' active configs differ from the master's.
> All the peers reported by the master and tablet servers are:
> A = 067e1e7245154f0fb2720dec6c77feec
> B = 43d450fd58794f008caca58d44a35ff8
> C = c4ed5cb73f5644a8804d3abc976d02f8
>
> The consensus matrix is:
> Config source | Voters | Current term | Config index | Committed?
> ---------------+--------------+--------------+--------------+------------
> master | A* B C | | | Yes
> A | A* B C | 12 | 7432 | Yes
> B | A B C | 11 | 7432 | Yes
> C | A* B C | 12 | 7432 | Yes
> Table hwx_log has 2 under-replicated tablet(s)
>
> the tablet cann't be add_replica or remove_replica, and I try to delete
> the data of current partition，error also occur：
> WARNINGS:
> Kudu error(s) reported, first error: Timed out: Failed to write batch of
> 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545
> attempt(s): Failed to write to server: (no server available): Write(tablet:
> 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545)
> passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8
> is not leader of this config. Role: FOLLOWER. Consensus state:
> current_term: 10 leader_uuid: "" committed_config { opid_index: 13049
> OBSOLETE_local: false peers { permanent_uuid: "
> ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index:
> 13692 OBSOLETE_local: false peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } }
>
>
>
> Kudu error(s) reported, first error: Timed out: Failed to write batch of
> 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545
> attempt(s): Failed to write to server: (no server available): Write(tablet:
> 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545)
> passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8
> is not leader of this config. Role: FOLLOWER. Consensus state:
> current_term: 10 leader_uuid: "" committed_config { opid_index: 13049
> OBSOLETE_local: false peers { permanent_uuid: "
> ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index:
> 13692 OBSOLETE_local: false peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } }
> Error in Kudu table 'hwx_log': Timed out: Failed to write batch of 76249
> ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s):
> Failed to write to server: (no server available): Write(tablet:
> 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545)
> passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8
> is not leader of this config. Role: FOLLOWER. Consensus state:
> current_term: 10 leader_uuid: "" committed_config { opid_index: 13049
> OBSOLETE_local: false peers { permanent_uuid: "
> ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index:
> 13692 OBSOLETE_local: false peers { permanent_uuid: "
> c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: "
> 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr {
> host: "cloud-ocean-kudu-04" port: 7050 } } } (1 of 76249 similar)
>
> 2017-09-06 14:04 GMT+08:00 Lee King <yu...@gmail.com>:
>
>> We got an error about :Service unavailable: Transaction failed, tablet
>> 2758e5c68e974b92a3060db8575f3621 transaction memory consumption
>> (67031036) has exceeded its limit (67108864) or the limit of an ancestral
>> tracker.It looks like https://issues.apache.org/jira/browse/KUDU-1912.
>> and the bug will be fix at 1.5,but out version is 1.4,Is there any affect
>> for kudu stablity or data consistency?
>>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Service unavailable: Transaction failed, tablet
2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036)
has exceeded its limit (67108864) or the limit of an ancestral tracker
Posted by Lee King <yu...@gmail.com>.
I use kudu command line tool cluster ksck, the error detail:
Fetched info from all 10 Tablet Servers
Tablet 2758e5c68e974b92a3060db8575f3621 of table 'hwx_log' is
under-replicated: 1 replica(s) not RUNNING
067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): bad state
State: FAILED
Data state: TABLET_DATA_READY
Last status: Service unavailable: Transaction failed, tablet
2758e5c68e974b92a3060db8575f3621 transaction memory consumption (67031036)
has exceeded its limit (67108864) or the limit of an ancestral tracker
ad1ea284caff4b07a705c9156b0811cd (cloud-ocean-kudu-01:7050): RUNNING
c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING
[LEADER]
3 replicas' active configs differ from the master's.
All the peers reported by the master and tablet servers are:
A = 067e1e7245154f0fb2720dec6c77feec
B = ad1ea284caff4b07a705c9156b0811cd
C = c4ed5cb73f5644a8804d3abc976d02f8
The consensus matrix is:
Config source | Voters | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 10 | 13049 | Yes
B | A B C | 10 | 13049 | Yes
C | A C | 10 | 13692 | No
Tablet 5f996309d53445769c7db89abaa05845 of table 'hwx_log' is
under-replicated: 1 replica(s) not RUNNING
067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): RUNNING
[LEADER]
43d450fd58794f008caca58d44a35ff8 (cloud-ocean-kudu-03:7050): bad state
State: FAILED
Data state: TABLET_DATA_READY
Last status: Service unavailable: Transaction failed, tablet
5f996309d53445769c7db89abaa05845 transaction memory consumption (66844571)
has exceeded its limit (67108864) or the limit of an ancestral tracker
c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING
1 replicas' active configs differ from the master's.
All the peers reported by the master and tablet servers are:
A = 067e1e7245154f0fb2720dec6c77feec
B = 43d450fd58794f008caca58d44a35ff8
C = c4ed5cb73f5644a8804d3abc976d02f8
The consensus matrix is:
Config source | Voters | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 12 | 7432 | Yes
B | A B C | 11 | 7432 | Yes
C | A* B C | 12 | 7432 | Yes
Table hwx_log has 2 under-replicated tablet(s)
the tablet cann't be add_replica or remove_replica, and I try to delete the
data of current partition，error also occur：
WARNINGS:
Kudu error(s) reported, first error: Timed out: Failed to write batch of
76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s):
Failed to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
Kudu error(s) reported, first error: Timed out: Failed to write batch of
76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s):
Failed to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
Error in Kudu table 'hwx_log': Timed out: Failed to write batch of 76249
ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): Failed
to write to server: (no server available): Write(tablet:
2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) passed
its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 is
not leader of this config. Role: FOLLOWER. Consensus state: current_term:
10 leader_uuid: "" committed_config { opid_index: 13049 OBSOLETE_local:
false peers { permanent_uuid: "ad1ea284caff4b07a705c9156b0811cd"
member_type: VOTER last_known_addr { host: "cloud-ocean-kudu-01" port: 7050
} } peers { permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type:
VOTER last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } }
pending_config { opid_index: 13692 OBSOLETE_local: false peers {
permanent_uuid: "c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-02" port: 7050 } } peers {
permanent_uuid: "067e1e7245154f0fb2720dec6c77feec" member_type: VOTER
last_known_addr { host: "cloud-ocean-kudu-04" port: 7050 } } } (1 of 76249
similar)
2017-09-06 14:04 GMT+08:00 Lee King <yu...@gmail.com>:
> We got an error about :Service unavailable: Transaction failed, tablet
> 2758e5c68e974b92a3060db8575f3621 transaction memory consumption
> (67031036) has exceeded its limit (67108864) or the limit of an ancestral
> tracker.It looks like https://issues.apache.org/jira/browse/KUDU-1912.
> and the bug will be fix at 1.5,but out version is 1.4,Is there any affect
> for kudu stablity or data consistency?
>