You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Jukka Zitting (JIRA)" <ji...@apache.org> on 2014/09/03 19:38:51 UTC
[jira] [Commented] (OAK-2070) Segment corruption
[ https://issues.apache.org/jira/browse/OAK-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120136#comment-14120136 ]
Jukka Zitting commented on OAK-2070:
------------------------------------
bq. The segment id looks wrong, my guess is that the segment header got corrupt somehow.
Sounds correct. All the UUIDs created by the SegmentMK are of the form {{xxxxxxxx-xxxx-4xxx-Txxx-xxxxxxxxxxxx}}, where {{T}} is either {{a}} or {{b}} depending on the type of the segment.
bq. I think the wrong byte sequence 00049b053927000498000493 is regular record data.
Yep, looks like that.
It sounds like there is a bug in the {{SegmentWriter.flush()}} logic somewhere around https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.0.5/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java#L182. When filling up a segment buffer, the SegmentWriter will write records backwards from the end of the buffer and the segment reference lookup table forwards from the beginning of the buffer. The buffer gets flushed into a persisted segment whenever these two areas would overlap or when other size limits would be reached (see [prepare()|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.0.5/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java#L267]). It might be that we're hitting some edge case that the current code doesn't handle correctly.
This is with latest Oak trunk, right?
> Segment corruption
> ------------------
>
> Key: OAK-2070
> URL: https://issues.apache.org/jira/browse/OAK-2070
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
> Reporter: Thomas Mueller
> Fix For: 1.1
>
> Attachments: SearchByteSeq.java, SegmentDump.java
>
>
> I got a strange case of a corrupt segment. The error message is "java.lang.IllegalStateException: Segment b6e0cdbc-0004-9b05-3927-000498000493 not found". The segment id looks wrong, my guess is that the segment header got corrupt somehow. The checksum of the entry is correct however. I wrote a simple segment header dump tool (I will attach it), which printed:
> {noformat}
> segmentId: b6fec1b4-f321-4b64-aeab-31c27a939287.4c65e101
> length: 40000
> maxRef: 10000
> checksum: 4c65e101
> magic: 0aK
> version: 0
> id count: 66
> root count: 24
> blob ref count: 0
> reserved: 0
> segment #0: e44fec74-3bc9-4cee-a864-5f5fb4485cb3
> segment #1: b6480852-9af2-4f21-a542-933cd554e9ed
> segment #2: bec8216a-64e3-4683-a28a-08fbd7ed91f9
> segment #3: b8067f5a-16af-429f-a1b5-c2f1ed1d3c59
> segment #4: 7c2e6167-b292-444a-a318-4f43692f40f2
> segment #5: 3b707f6e-d1e8-4bc0-ace5-67499ac4c369
> segment #6: 0c15bf0a-770d-4057-af6b-baeaeffde613
> segment #7: 97814ad5-3fc3-436e-a7c6-2aed13bcf529
> segment #8: b4babf94-588f-4b32-a8d6-08ba3351abf2
> segment #9: 2eab77d7-ee41-4519-ad14-c31f4f72d2e2
> segment #10: 57949f56-15d1-4f65-a107-d517a222cb11
> segment #11: d25bb0f8-3948-4d33-a35a-50a60a1e9f27
> segment #12: 58066744-e2d6-4f1c-a962-caae337f48db
> segment #13: b195e811-a984-4b3d-a4c8-5fb33572bfce
> segment #14: 384b360d-69ce-4df4-a419-65b80d39b40c
> segment #15: 325b6083-de56-4a4b-a6e5-2eac83e9d4aa
> segment #16: e3839e53-c516-4dce-a39d-b6241d47ef1d
> segment #17: a56f0a75-625b-4684-af43-9936d4d0c4bd
> segment #18: b7428409-8dc0-4c5b-a050-78c838db0ef1
> segment #19: 6ccd5436-37a1-4b46-a5a4-63f939f2de76
> segment #20: 4f28ca6e-a52b-4a06-acb0-19b710366c96
> segment #21: 137ec136-bcfb-405b-a5e3-59b37c54229c
> segment #22: 8a53ffe4-6941-4b9b-a517-9ab59acf9b63
> segment #23: effd3f42-dff3-4082-a0fc-493beabe8d42
> segment #24: 173ef78d-9f41-49fb-ac25-70daba943b37
> segment #25: 9d3ab04d-d540-4256-a28f-c71134d1bfd0
> segment #26: d1966946-45c7-4539-a8f2-9c7979b698c5
> segment #27: b2a72be0-ee4e-4f9d-a957-f84908f05040
> segment #28: 8eead39e-62f7-4b6c-ac12-5caee295b6e5
> segment #29: 19832493-415c-46d2-aa43-0dd95982d312
> segment #30: 462cd84b-3638-4c8c-a273-42b1be5d3df3
> segment #31: 4b4f6a2d-1281-475c-adbe-34457b6de60d
> segment #32: e78b13e2-039d-4f73-a3a8-d25da3b78a93
> segment #33: 03b652ac-623b-4d3d-a5f0-2e7dcd23fd88
> segment #34: 3ff1c4cf-cbd7-45bb-a23a-7e033c64fe87
> segment #35: ed9fb9ad-0a8c-4d37-ab95-c2211cde6fee
> segment #36: 505f766d-509d-4f1b-ad0b-dfd59ef6b6a1
> segment #37: 398f1e80-e818-41b7-a055-ed6255f6e01b
> segment #38: 74cd7875-df17-43af-a2cc-f3256b2213b9
> segment #39: ad884c29-61c9-4bd5-ad57-76ffa32ae3e8
> segment #40: 29cfb8c8-f759-41b2-ac71-d35eac0910ad
> segment #41: af5032e1-b473-47ad-a8e2-a17e770982e8
> segment #42: 37263fa3-3331-4e89-af1f-38972b65e058
> segment #43: 81baf70d-5529-416f-a8bd-9858d62fd2cc
> segment #44: aa3abdfb-4fc7-4c49-a628-fb097f8db872
> segment #45: 81e958b4-0493-4a7b-ab92-5f3cc13119e7
> segment #46: f03d0200-fe8e-4e93-a94a-97b9052f13ab
> segment #47: d9931e67-cc8c-4f45-a1b5-129a0b07e1fb
> segment #48: e32ad441-12d9-4263-a258-b81bd85f499e
> segment #49: 2093a577-e797-44b5-aa3e-45c08b1ad9c1
> segment #50: dadf47d7-9728-43dd-aa55-177ccbc45b81
> segment #51: 9db4b6f5-c21c-4e27-a4ab-9a0c419477b2
> segment #52: 2ca39d68-e7bf-4c83-aebe-f708fb23659e
> segment #53: 5dc42abe-e17b-41a3-a890-47c83eeee19a
> segment #54: a817c7fb-4f05-429e-ab23-49d68e222e6c
> segment #55: 8a5de869-a8a8-42cb-ac78-ec22763fe558
> segment #56: cca0c5d8-9269-4b1f-ad3b-1ca666ca9cb9
> segment #57: ff299dda-00ce-4e01-ac1c-2289447500b7
> segment #58: 0c94d46e-7fae-474f-ac89-8ba87a64cb56
> segment #59: 890f7fcc-7c7e-4dc7-acc5-ed0927ef1a9f
> segment #60: 16f5de56-2aaa-4ca2-afa7-4bcb536b74b1
> segment #61: 97db1991-1c29-4f8c-ae05-7f79b712fb4c
> segment #62: 936aa29a-2679-4998-a159-d9a19069786a
> segment #63: 0d1abedd-8c20-4c1b-a9c8-f60d7cb94008
> segment #64: 1ffb6ac4-d7a9-44c5-aaa6-bf967e87dec0
> segment #65: b6e0cdbc-0004-9b05-3927-000498000493
> ERROR: not a segment?
> root record reference #0: 7766a
> root record reference #1: 76f93
> root record reference #2: 75e72
> root record reference #3: 758c6
> root record reference #4: 7529a
> root record reference #5: 74c59
> root record reference #6: 7464f
> root record reference #7: 740e0
> root record reference #8: 73ae7
> root record reference #9: 734be
> root record reference #10: 7274d
> root record reference #11: 7210e
> root record reference #12: 71c3e
> root record reference #13: 7166d
> root record reference #14: 710e1
> root record reference #15: 70b29
> root record reference #16: 70647
> root record reference #17: 70644
> root record reference #18: 7063d
> root record reference #19: 70636
> root record reference #20: 7062f
> root record reference #21: 70629
> root record reference #22: 70622
> root record reference #23: 70109
> {noformat}
> So just the last segment reference seems to be broken. The segment above is entry #250 in that file. The referenced segments in that segment refer to slightly older segments in the same file. The broken reference starts with b6e0cdbc, this seems to be intact. The reference seems to point to entry #242 in that file (a bit earlier).
> {noformat}
> ...
> segment #61: 97db1991-1c29-4f8c-ae05-7f79b712fb4c -> entry #220
> segment #62: 936aa29a-2679-4998-a159-d9a19069786a -> entry #241
> segment #63: 0d1abedd-8c20-4c1b-a9c8-f60d7cb94008 -> entry #210
> segment #64: 1ffb6ac4-d7a9-44c5-aaa6-bf967e87dec0 -> entry #228
> segment #65: b6e0cdbc-0004-9b05-3927-000498000493 -> entry #242?
> -> would be b6e0cdbc-b5cf-494f-adfa-9dad38944b8d
> {noformat}
> I think the wrong byte sequence 00049b053927000498000493 is regular record data. I found the pattern "00049b053927" in a later segment (entry #277 in the same file), and segment data in general looks like this (many zeroes, many 0004 sequences).
> So far I don't know what could have caused the corruption. It is strange that the same byte pattern appears later in the same file, but maybe it's just a common byte pattern.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)