You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Juan Yu (JIRA)" <ji...@apache.org> on 2017/01/03 17:45:58 UTC
[jira] [Commented] (KUDU-1768) Tablet server cannot recover after
run out of disk space
[ https://issues.apache.org/jira/browse/KUDU-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795631#comment-15795631 ]
Juan Yu commented on KUDU-1768:
-------------------------------
BTW, It can also crash.
{code}
Log file created at: 2016/12/30 20:16:58
Running on machine: impala4135-2.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F1230 20:16:58.748709 30089 tablet_peer_mm_ops.cc:128] Check failed: _s.ok() FlushMRS failed on e8b5861a24fb4ada9bf2e219b87b73ef: IO error: Failed to open DiskRowSet for flush: Unable to Start() writer for column double_col181[double NULLABLE]: Couldn't write header: /dataroot/dataroot/kudu/data/data/b0096794642b40c1b0ff7587d5082b7c.data: No space left on device (error 28)
{code}
> Tablet server cannot recover after run out of disk space
> --------------------------------------------------------
>
> Key: KUDU-1768
> URL: https://issues.apache.org/jira/browse/KUDU-1768
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Juan Yu
> Attachments: tserver.log.tar.gz
>
>
> My node has small disk. after upsert into a large table several times, it ran out of disk space. I dropped the table. all nodes dropped the tablet successfully except one. it stuck in a bad state and tablet data are not removed even after restart.
> For dropped table, it shows
> {code}
> tmp_store_sales d88f8a7eba974cba8d9c547f69168c6b hash buckets: (0) FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first
> {code}
> for other tables, it shows
> {code}
> kudu_store_sales f539ce77657e431aa5ce4ab7f7375f84 hash buckets: (0, 7) FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28) IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)
> {code}
> attached tablet server log from this node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)