You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2019/03/26 15:56:59 UTC
[GitHub] [incubator-openwhisk] sven-lange-last commented on issue #4357:
Test cases with consistent failures in Jenkins pipeline build
sven-lange-last commented on issue #4357: Test cases with consistent failures in Jenkins pipeline build
URL: https://github.com/apache/incubator-openwhisk/issues/4357#issuecomment-476717350
The failing test is meant to check whether the specified memory limit is properly enforced when running an action. This enforcement is actually not performed by the invoker or Docker - but by the memory cgroup controller. The invoker passes the desired memory limit to Docker which, in turn, creates proper limits in the memory cgroup hierarchy for the action container.
Output from a failing test:
```
org.apache.openwhisk.core.limits.ActionLimitsTests > Action limits should be aborted when exceeding its memory limits STANDARD_OUT
check failed for activation 5ebe89734f344198be89734f3441986b: {"activationId":"5ebe89734f344198be89734f3441986b","annotations":[{"key":"path","value":"guest/TestNodeJsMemoryExceeding"},{"key":"waitTime","value":1938},{"key":"kind","value":"nodejs:6"},{"key":"timeout","value":false},{"key":"limits","value":{"concurrency":1,"logs":10,"memory":128,"timeout":60000}},{"key":"initTime","value":161}],"duration":3562,"end":1553612871065,"logs":["2019-03-26T15:07:47.670298448Z stdout: helloEatMemory: memory 256MB","2019-03-26T15:07:47.680340352Z stdout: helloEatMemory: memoryInB=268435456, memoryInPages=65536","2019-03-26T15:07:47.680377547Z stdout: helloEatMemory: pagesPerBlock=8, blockSizeInB=32768, memoryInBlocks=8192","2019-03-26T15:07:50.418313718Z stdout: helloEatMemory: blocks.length=8192","2019-03-26T15:07:50.418342259Z stdout: helloEatMemory: completed allocating memory","2019-03-26T15:07:50.43882444Z stdout: { rss: 119693312,","2019-03-26T15:07:50.438836755Z stdout: heapTotal: 21004288,","2019-03-26T15:07:50.438840614Z stdout: heapUsed: 10310224,","2019-03-26T15:07:50.43884384Z stdout: external: 268452540 }"],"name":"TestNodeJsMemoryExceeding","namespace":"guest","publish":false,"response":{"result":{"msg":"OK, buffer of size 256 MB has been filled."},"status":"success","success":true},"start":1553612867503,"subject":"guest","version":"0.0.1"}
Exception occurred during test execution: java.util.NoSuchElementException: key not found: error
```
The test fails because the action completes successfully instead of being stopped due to exceeding its memory limit.
I see two possible reasons:
1. The system on which the action container is running does not support memory limits.
2. The test action does not exceed the memory limit because memory management works in an unexpected way.
#### Missing memory limit support
@houshengbo please run the `docker info` command on the system. Can you please paste the output to this issue? Does Docker warn about missing features?
Follow Docker's troubleshooting procedure: https://docs.docker.com/install/linux/linux-postinstall/#troubleshooting. Namely, please run script `https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh` and paste the output to this issue. Is there any support missing around memory cgroups?
#### Test does not exceed memory limit
The test output from above contains `rss: 119693312`. I would expected a much higher value... According to the output, the action code is expected to allocate 8192 blocks of `Uint8Array` with size 32768. This should be 256 MiB - exceeding the limit of 128 MiB RSS. Probably, the action code does not lead to enough RSS consumption. Maybe the system uses memory compression or other techniques to save RSS?
Maybe we need to adapt following code portion to write more random numbers to memory pages? Or make sure that every single page in each allocated block is really updated with random numbers to prevent compression?
https://github.com/apache/incubator-openwhisk/blob/aad27bed182f709fd7d25bc7a01f7e0fe985b027/tests/dat/actions/memoryWithGC.js#L19-L31
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services