You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2016/12/04 17:51:58 UTC
[jira] [Commented] (KUDU-1768) Tablet server cannot recover after
run out of disk space
[ https://issues.apache.org/jira/browse/KUDU-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15720318#comment-15720318 ]
Jean-Daniel Cryans commented on KUDU-1768:
------------------------------------------
Looks like a dup of KUDU-1607. [~dineshabbi], any advice on fixing this?
> Tablet server cannot recover after run out of disk space
> --------------------------------------------------------
>
> Key: KUDU-1768
> URL: https://issues.apache.org/jira/browse/KUDU-1768
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Juan Yu
> Attachments: tserver.log.tar.gz
>
>
> My node has small disk. after upsert into a large table several times, it ran out of disk space. I dropped the table. all nodes dropped the tablet successfully except one. it stuck in a bad state and tablet data are not removed even after restart.
> For dropped table, it shows
> {code}
> tmp_store_sales d88f8a7eba974cba8d9c547f69168c6b hash buckets: (0) FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first
> {code}
> for other tables, it shows
> {code}
> kudu_store_sales f539ce77657e431aa5ce4ab7f7375f84 hash buckets: (0, 7) FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28) IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)
> {code}
> attached tablet server log from this node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)