[GitHub] dgrove-oss opened a new issue #3446: KubernetesInvokerAgentLogStore doesn't support logrotation

[GitBox <gi...@apache.org>]

dgrove-oss opened a new issue #3446: KubernetesInvokerAgentLogStore doesn't support logrotation
URL: https://github.com/apache/incubator-openwhisk/issues/3446
 
 
   ## Environment details:
   
   - Kubernetes (v 1.7 on IBM Cloud)
   - OpenWhisk invoker deployed with invoker.config.java_opts containing `-Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider -Dwhisk.spi.LogStoreProvider=whisk.core.containerpool.kubernetes.KubernetesInvokerAgentLogStoreProvider`
   
   ## Steps to reproduce the issue:
   
   1.   Define an action that generates significant volume of log output.  
   2.   Execute action multiple times until > 100Mb of log is generated
   3.   Log extraction will fail for the action invocation that crossed 100Mb of log output because of log rotation policy used when creating the container.  As a result, the container will be torn down and user logs will be lost.
   
   ## Analysis of Problem.
   Containers are being created with with docker's json-logdriver configured to perform log rotation.  This is desirable in that it limits the size of logfiles, however it breaks our LogProvider implementations that directly read the docker logs and store the 'lastOffset' read to allow extraction of just a single activation's log from the container's log stream.
   
   Using docker inspect on an action container, we can see the log configuration:
   
   / # docker inspect 735a0b5ea684
   ```
   [
       {
           "Id": "735a0b5ea6840736e0c700a4f02f8725893374e33f21b677a7dddad01357305d",
            ...... 
           "LogPath": "/var/lib/docker/containers/735a0b5ea6840736e0c700a4f02f8725893374e33f21b677a7dddad01357305d/735a0b5ea6840736e0c700a4f02f8725893374e33f21b677a7dddad01357305d-json.log",
            ......  
               "LogConfig": {
                   "Type": "json-file",
                   "Config": {
                       "max-file": "3",
                       "max-size": "100m"
                   }
               },
            ...... 
   ```
   
   It appears that Kubernetes intentionally does not expose the docker --log-opt flag on a per-container basis.  So a quick fix of simply doing`--log-opt max-size=-1` to disable log rotation is not possible.    Arguably, this fix is also not desirable since we should limit the size of container logs to avoid exhausting disk space on the worker node. 
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services