You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Mike Percy (JIRA)" <ji...@apache.org> on 2016/02/26 12:37:18 UTC
[jira] [Updated] (KUDU-411) Move the consensus RPCs out of
tserver_service
[ https://issues.apache.org/jira/browse/KUDU-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated KUDU-411:
----------------------------
Parent: KUDU-410
> Move the consensus RPCs out of tserver_service
> ----------------------------------------------
>
> Key: KUDU-411
> URL: https://issues.apache.org/jira/browse/KUDU-411
> Project: Kudu
> Issue Type: Sub-task
> Components: tablet
> Affects Versions: M4.5
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
>
> I tried creating a 150 tablets table with consensus set to 3 replicas and I see the following:
> {noformat}
> I0729 11:37:58.501868 13007 raft_consensus.cc:168] T 3d57b78c37464f2091a4be3d05383653 P 8039032e27344e1d90a3db4b7a124781 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=0): replicating to peers...
> W0729 11:37:58.513223 13000 consensus_peers.cc:390] Couldn't send request to peer 44696db5501b4ee0b6b233c90e2d8f88 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Remote error: Service unavailable: UpdateConsensus request on kudu.tserver.TabletServerService from 10.20.188.124:52904 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1 times.
> W0729 11:37:59.502914 13000 consensus_peers.cc:390] Couldn't send request to peer a64e2f0c055d4d62a6cb88bdf5ec03c3 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times.
> W0729 11:38:00.002295 13000 consensus_peers.cc:390] Couldn't send request to peer 44696db5501b4ee0b6b233c90e2d8f88 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times.
> W0729 11:38:00.003197 13000 consensus_peers.cc:390] Couldn't send request to peer a64e2f0c055d4d62a6cb88bdf5ec03c3 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Remote error: Service unavailable: UpdateConsensus request on kudu.tserver.TabletServerService from 10.20.188.124:42067 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 2 times.
> ...
> W0729 11:50:35.821658 13000 consensus_peers.cc:390] Couldn't send request to peer a64e2f0c055d4d62a6cb88bdf5ec03c3 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Remote error: Service unavailable: UpdateConsensus request on kudu.tserver.TabletServerService from 10.20.188.124:42067 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1513 times.
> W0729 11:50:35.821877 13000 consensus_peers.cc:390] Couldn't send request to peer 44696db5501b4ee0b6b233c90e2d8f88 for tablet 3d57b78c37464f2091a4be3d05383653: Status: Remote error: Service unavailable: UpdateConsensus request on kudu.tserver.TabletServerService from 10.20.188.124:52904 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1513 times.
> {noformat}
> All the TS are trying to talk to each other and everything's full. At the same time, the master starts creating new tablets because these ones aren't being created within the given timeout. It's also interesting to see that the tablet is still alive after 13 minutes when it was deleted at minute after its creation:
> {noformat}
> W0729 11:39:00.035020 7432 catalog_manager.cc:1573] Tablet 3d57b78c37464f2091a4be3d05383653 (table lineitem [id=5b1b20d435674108930d47b1061cdf7d]) was not created within the allowed timeout. Replacing with a new tablet 0fc88a3798e64433bf76e8f9689617cc
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)