[jira] [Commented] (OAK-9095) MapRecord corruption when adding more than MapRecord.MAX_SIZE entries in branch record

["Thomas Mueller (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118640#comment-17118640 ] 

Thomas Mueller commented on OAK-9095:
-------------------------------------

We need to document the limit, but not sure where to do that best.

About logging: It is quite tricky to avoid filling the log file (which might fill the disk), but get peoples attention...I think we should ensure that it won't fill the log file, but is detected by monitoring. What about:

* Log a warning (message A) at 400 million entries, but at most once per second. 
* Log an error (message B) at 450 million entries, but at most once per second. 
* If more than 500 million entries, only allow writes if the system property "oak.segmentNodeStore.allowWritesOnHugeMapRecord" is set. Not use a static field for this, but check the system property for each write operation, so that it's possible to set the system property on a running system without having to restart. If the system property is not set, an message B is logged and an exception is thrown, preventing this write.
* If more than 536 million entries, log message C and throw an exception even if the system property is set.

Message A: "Node <path> has more than 400 million direct child nodes. Please remove entries."
Message B: "Node <path> has more than 450 million direct child nodes. Writing more than 500 million nodes (up to the hard limit of 536 million) is only allowed if the system property "oak.segmentNodeStore.allowWritesOnHugeMapRecord" is set. 
Message C: "Node <path> has more than 536 million direct child nodes. Writing is not allowed. Please remove entries."

> MapRecord corruption when adding more than MapRecord.MAX_SIZE entries in branch record
> --------------------------------------------------------------------------------------
>
>                 Key: OAK-9095
>                 URL: https://issues.apache.org/jira/browse/OAK-9095
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>    Affects Versions: 1.22.3, 1.8.22, 1.30.0
>            Reporter: Andrei Dulceanu
>            Assignee: Andrei Dulceanu
>            Priority: Major
>
> It is now possible to write a {{MapRecord}} with a huge number of entries, going over the maximum limit, {{MapRecord.MAX_SIZE}}, (i.e. 536.870.911 entries). This issue stems from the fact that the number of entries is checked when writing a map leaf record [0], but not when writing a map branch record [1]. When more than {{MapRecord.MAX_SIZE}} entries are written in a branch record [2], the {{entrycCount}} overflows in the first bit of the level, essentially rendering the entire HAMT structure corrupt, since the root branch record will be stored now at level 1, instead of level 0, reporting an incorrect size as well (i.e. actual size - {{MapRecord.MAX_SIZE}}).
> Since this is a hard limit of the segment store and going above this number would mean rewriting the internals of the HAMT structure currently in use, I propose the following mitigation:
> * add a size check for the branch record to not allow going over the limit
> * log a warning when the number of entries goes over 400.000.000
> * log an error when the number of entries goes over 500.000.000 and do not allow any write operations on the node
> * allow further writes only if {{oak.segmentNodeStore.allowWritesOnHugeMapRecord}} system property is present
> [0] https://github.com/apache/jackrabbit-oak/blob/1.22/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java#L284
> [1] https://github.com/apache/jackrabbit-oak/blob/1.22/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/DefaultSegmentWriter.java#L291
> [2] https://github.com/apache/jackrabbit-oak/blob/1.22/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/RecordWriters.java#L231



--
This message was sent by Atlassian Jira
(v8.3.4#803005)