You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Juan Yu (JIRA)" <ji...@apache.org> on 2017/01/03 17:45:58 UTC

[jira] [Commented] (KUDU-1768) Tablet server cannot recover after run out of disk space

    [ https://issues.apache.org/jira/browse/KUDU-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795631#comment-15795631 ] 

Juan Yu commented on KUDU-1768:
-------------------------------

BTW, It can also crash.
{code}
Log file created at: 2016/12/30 20:16:58
Running on machine: impala4135-2.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F1230 20:16:58.748709 30089 tablet_peer_mm_ops.cc:128] Check failed: _s.ok() FlushMRS failed on e8b5861a24fb4ada9bf2e219b87b73ef: IO error: Failed to open DiskRowSet for flush: Unable to Start() writer for column double_col181[double NULLABLE]: Couldn't write header: /dataroot/dataroot/kudu/data/data/b0096794642b40c1b0ff7587d5082b7c.data: No space left on device (error 28)
{code}

> Tablet server cannot recover after run out of disk space
> --------------------------------------------------------
>
>                 Key: KUDU-1768
>                 URL: https://issues.apache.org/jira/browse/KUDU-1768
>             Project: Kudu
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Juan Yu
>         Attachments: tserver.log.tar.gz
>
>
> My node has small disk. after upsert into a large table several times, it ran out of disk space. I dropped the table. all nodes dropped the tablet successfully except one. it stuck in a bad state and tablet data are not removed even after restart.
> For dropped table, it shows 
> {code}
> tmp_store_sales	d88f8a7eba974cba8d9c547f69168c6b	hash buckets: (0)	FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first			Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first
> {code}
> for other tables, it shows
> {code}
> kudu_store_sales	f539ce77657e431aa5ce4ab7f7375f84	hash buckets: (0, 7)	FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)			IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)
> {code}
> attached tablet server log from this node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)