You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "David Snowsill (Jira)" <ji...@apache.org> on 2020/06/15 18:47:00 UTC
[jira] [Comment Edited] (OAK-9112) High CPU IOWAIT on Mongo due to
Oak background task
[ https://issues.apache.org/jira/browse/OAK-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136099#comment-17136099 ]
David Snowsill edited comment on OAK-9112 at 6/15/20, 6:46 PM:
---------------------------------------------------------------
Hi Marcel, thanks for your help.
I can confirm that over the period the background process was running it was iterating through the nodes collection performing queries such as:
{code}
{ desc: "conn1069", threadId: "140061220099840", connectionId: 1069, client: "18.207.114.70:53082", clientMetadata: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 390894, secs_running: 0, microsecs_running: { "$numberLong": "187909" }, op: "query", ns: "prod-content-service.nodes", query: { find: "nodes", filter: { _id: "2:/prod/c79fc74d-99b8-45b2-a83e-961158cb64d9" }, limit: 1, singleBatch: true }, planSummary: "IDHACK", numYields: 0, locks: { Global: "r", Database: "r", Collection: "r" }, waitingForLock: false, lockStats: { Global: { acquireCount: { r: { "$numberLong": "2" } } }, Database: { acquireCount: { r: { "$numberLong": "1" } } }, Collection: { acquireCount: { r: { "$numberLong": "1" } } } } }
{code}
with the only parameter changing in the query being the "_id"
The next time it runs we will try to profile it and provide a stack trace
Regards,
David
was (Author: djsnowsill):
Hi Marcel, thanks for your help.
I can confirm that over the period the background process was running it was iterating through the nodes collection performing queries such as:
{code:java}
{ desc: "conn1069", threadId: "140061220099840", connectionId: 1069, client: "18.207.114.70:53082", clientMetadata: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 390894, secs_running: 0, microsecs_running: { "$numberLong": "187909" }, op: "query", ns: "prod-content-service.nodes", query: { find: "nodes", filter: { _id: "2:/prod/c79fc74d-99b8-45b2-a83e-961158cb64d9" }, limit: 1, singleBatch: true }, planSummary: "IDHACK", numYields: 0, locks: { Global: "r", Database: "r", Collection: "r" }, waitingForLock: false, lockStats: { Global: { acquireCount: { r: { "$numberLong": "2" } } }, Database: { acquireCount: { r: { "$numberLong": "1" } } }, Collection: { acquireCount: { r: { "$numberLong": "1" } } } } }
{code:java}
with the only parameter changing in the query being the "_id"
The next time it runs we will try to profile it and provide a stack trace
Regards,
David
> High CPU IOWAIT on Mongo due to Oak background task
> ---------------------------------------------------
>
> Key: OAK-9112
> URL: https://issues.apache.org/jira/browse/OAK-9112
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: mongomk
> Affects Versions: 1.8.22
> Environment: Mongo: 3.4.9
> Hosteding: Scalegrid
> Instance Size: Small
> Reporter: David Snowsill
> Priority: Major
> Fix For: 1.8.22
>
> Attachments: image-2020-06-15-19-41-04-852.png, image-2020-06-15-19-41-40-385.png
>
>
> Over the previous weekend, our existing application went from having 100,000 documents in the "nodes" collection to 600,000 documents in the "nodes" collection. Since that time there appears to be an Oak background task that commences running near 100% CPU IOWAIT and heavy disk activity (read only). Please refer to charts below
> !image-2020-06-15-19-41-04-852.png!
> !image-2020-06-15-19-41-40-385.png!
> According to currentOp Mongo command, the queries being running look as follows:
> { desc: "conn4944", threadId: "140061124310784", connectionId: 4944, client: "18.207.114.70:47008", clientMetadata: \{ driver: { name: "mongo-java-driver", version: "unknown" }, os: \{ type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 2215767, secs_running: 2, microsecs_running: \{ "$numberLong": "2280333" }, op: "query", ns: "prod-content-service.nodes", query: \{ find: "nodes", filter: { _id: "2:/prod/18d6b4d0-9ee8-4a06-804e-b3669f67dda8" }, limit: 1, singleBatch: true }, planSummary: "IDHACK", numYields: 0, locks: \{ Global: "r", Database: "r", Collection: "r" }, waitingForLock: false, lockStats: \{ Global: { acquireCount: { r: { "$numberLong": "2" } } }, Database: \{ acquireCount: { r: { "$numberLong": "1" } } }, Collection: \{ acquireCount: { r: { "$numberLong": "1" } } } } }
> and appears to be iterating over the entire collection.
> Can anyone provide any guidance on what is happening?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)