[jira] [Commented] (KUDU-1484) kudu-tserver flush failed during compaction, which cause data corruption.

["Adar Dembo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328412#comment-15328412 ] 

Adar Dembo commented on KUDU-1484:
----------------------------------

[~bruceSz] have you seen errors like this on other tservers? If not, we should treat this as an isolated incident of corruption and start looking at obvious culprits like the disks. Could you check whether the machine in question has any errors in the kernel log that suggest a disk (or disk controller) failure of some sort? Also, if you've only seen this on the one tserver, could you look at earlier logs for other interesting or related errors?

Finally, if all of the affected tablets are replicated at least three times (which they should be by default), you should be able to take the tserver offline and make progress with the remaining machines in your cluster. After you do that, the master should notice the underreplication and create extra replicas as needed (it may be good to add more capacity to the cluster).

> kudu-tserver flush failed  during compaction, which cause data corruption.
> --------------------------------------------------------------------------
>
>                 Key: KUDU-1484
>                 URL: https://issues.apache.org/jira/browse/KUDU-1484
>             Project: Kudu
>          Issue Type: Bug
>            Reporter: zhangsong
>            Priority: Critical
>
> foun some warning message in one of my kudu-tserver:
> W0613 15:07:27.300922 40713 tablet.cc:954] Compaction failed on 87cd1b5a25344d48a9cecc150bfe4689: Corruption: Flush to disk failed: offsets_pos 4278517760 > block size 46 in plain string block
> meanwhile , when issue query against related table (using impala), impala throw such error message: "Unable to advance iterator: Corruption: offsets_pos 4280811520 > block size 108 in plain string block
> "
> this has make the whole table unavailable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)