You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "xiaokai.wang (JIRA)" <ji...@apache.org> on 2019/06/22 01:23:00 UTC
[jira] [Comment Edited] (KUDU-2635) Tserver crash because some
orphaned blocks are still listed when deleting metadata
[ https://issues.apache.org/jira/browse/KUDU-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869714#comment-16869714 ]
xiaokai.wang edited comment on KUDU-2635 at 6/22/19 1:22 AM:
-------------------------------------------------------------
[~andrew.wong]
dmesg info:
{{[22989788.224136] blk_partition_remap: fail for partition 1 }}{{[22989788.224220] blk_partition_remap: fail for partition 1 }}{{[22989788.224304] blk_partition_remap: fail for partition 1 }}{{[22990224.699192] blk_partition_remap: fail for partition 1 }}{{[22990224.699380] EXT4-fs error: 1570 callbacks suppressed }}{{[22990224.699381] EXT4-fs error (device nvme8n1p1): ext4_find_entry:1469: inode #63045636: comm kudu-tserver: reading directory lblock 0 }}{{[22990224.699761] EXT4-fs: 1570 callbacks suppressed }}{{[22990224.699762] EXT4-fs (nvme8n1p1): previous I/O error to superblock detected }}{{[22990224.700117] blk_partition_remap: fail for partition 1 }}{{[22990224.700303] buffer_io_error: 1570 callbacks suppressed }}{{[22990224.700304] Buffer I/O error on dev nvme8n1p1, logical block 0, lost sync page write }}{{[22990224.702667] blk_partition_remap: fail for partition 1 }}{{[22990224.702901] blk_partition_remap: fail for partition 1 }}{{[22991655.443252] blk_partition_remap: fail for partition 1 }}{{[22991655.443585] EXT4-fs warning (device nvme8n1p1): htree_dirblock_to_tree:962: inode #63045635: lblock 0: comm ls: error -5 reading directory block}}
But the kudu-tserver restart successfully.
was (Author: xiaokai):
[~andrew.wong]
dmesg info:
[22989788.224136] blk_partition_remap: fail for partition 1 [22989788.224220] blk_partition_remap: fail for partition 1 [22989788.224304] blk_partition_remap: fail for partition 1 [22990224.699192] blk_partition_remap: fail for partition 1 [22990224.699380] EXT4-fs error: 1570 callbacks suppressed [22990224.699381] EXT4-fs error (device nvme8n1p1): ext4_find_entry:1469: inode #63045636: comm kudu-tserver: reading directory lblock 0 [22990224.699761] EXT4-fs: 1570 callbacks suppressed [22990224.699762] EXT4-fs (nvme8n1p1): previous I/O error to superblock detected [22990224.700117] blk_partition_remap: fail for partition 1 [22990224.700303] buffer_io_error: 1570 callbacks suppressed [22990224.700304] Buffer I/O error on dev nvme8n1p1, logical block 0, lost sync page write [22990224.702667] blk_partition_remap: fail for partition 1 [22990224.702901] blk_partition_remap: fail for partition 1 [22991655.443252] blk_partition_remap: fail for partition 1 [22991655.443585] EXT4-fs warning (device nvme8n1p1): htree_dirblock_to_tree:962: inode #63045635: lblock 0: comm ls: error -5 reading directory block
But the kudu-tserver restart successfully.
> Tserver crash because some orphaned blocks are still listed when deleting metadata
> ----------------------------------------------------------------------------------
>
> Key: KUDU-2635
> URL: https://issues.apache.org/jira/browse/KUDU-2635
> Project: Kudu
> Issue Type: Bug
> Components: fs, tablet, tserver
> Affects Versions: 1.7.0
> Reporter: Andrew Wong
> Assignee: Andrew Wong
> Priority: Major
>
> In some cases, upon deleting a tablet, a tablet server may fail to delete some blocks, and then fail to delete the tablet metadata, leading to a crash since failure to delete metadata is a fatal error. That's what happened in the below logs, but it's unclear why the blocks failed to be deleted, and why the server stayed up for a couple minutes after before receiving a delete tablet request, and ultimately crashing. Following the crash, the server was able to start up successfully.
>
> {{I1130 00:00:07.565915 29721 tablet_service.cc:795] Processing DeleteTablet for tablet 1db7aa7e81474907ace3d493c24cdc94 with delete_type TABLET_DATA_DELETED (Partition dropped at 2018-11-30 00:00:07 PST) from \{username='kudu'} at 10.93.87.15:47194}}
> {{I1130 00:00:07.565929 29721 tablet_replica.cc:262] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc: stopping tablet replica}}
> {{I1130 00:00:07.565954 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op CompactRowSetsOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.565997 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op MinorDeltaCompactionOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566010 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op MajorDeltaCompactionOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566020 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op UndoDeltaBlockGCOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566032 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op FlushMRSOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566040 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op FlushDeltaMemStoresOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566048 29721 maintenance_manager.cc:235] P 97235196a93b41c29954ed8534aa2ddc: Unregistered op LogGCOp(1db7aa7e81474907ace3d493c24cdc94)}}
> {{I1130 00:00:07.566056 29721 raft_consensus.cc:2012] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc [term 3 FOLLOWER]: Raft consensus shutting down.}}
> {{I1130 00:00:07.566074 29721 raft_consensus.cc:2039] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc [term 3 FOLLOWER]: Raft consensus is shut down!}}
> {{I1130 00:00:07.666061 29721 ts_tablet_manager.cc:1277] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc: Deleting tablet data with delete state TABLET_DATA_DELETED}}
> {{I1130 00:00:08.102607 29721 ts_tablet_manager.cc:1290] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.1166195}}
> {{I1130 00:00:08.102629 29721 log.cc:981] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc: Deleting WAL directory at /home/kudu/tablet/wal/wals/1db7aa7e81474907ace3d493c24cdc94}}
> {{I1130 00:00:08.103217 29721 ts_tablet_manager.cc:1310] T 1db7aa7e81474907ace3d493c24cdc94 P 97235196a93b41c29954ed8534aa2ddc: Deleting consensus metadata}}
> {{F1130 00:00:08.155643 29721 ts_tablet_manager.cc:848] Failed to delete tablet data for 1db7aa7e81474907ace3d493c24cdc94: Invalid argument: Unable to delete on-disk data from tablet 1db7aa7e81474907ace3d493c24cdc94: The metadata for tablet 1db7aa7e81474907ace3d493c24cdc94 still references orphaned blocks. Call DeleteTabletData() first}}
> {{I1130 00:02:09.460352 29725 tablet_service.cc:795] Processing DeleteTablet for tablet 1db7aa7e81474907ace3d493c24cdc94 with delete_type TABLET_DATA_DELETED (Partition dropped at 2018-11-30 00:00:07 PST) from \{username='kudu'} at 10.93.87.15:47194}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)