You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Ole Ersoy <ol...@gmail.com> on 2018/04/03 18:44:14 UTC

Container Log Collection for Microservices

I accidentally deleted the original thread, but saw that there were some questions surrounding logging to stdout (I assume while running in a microservice dockerized environment).

You might find these article helpful:
http://callistaenterprise.se/blogg/teknik/2017/07/29/building-microservices-part-7-distributed-tracing/
http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/

These cover log event collection for both ELK and Zipkin.  Parts 1-6 in the series are really good as well if you want to know more about microservices and security in general with the Spring and Netflix OSS stack.  Originally came across it while looking for OAuth material.

Cheers,
Ole

Re: Container Log Collection for Microservices

Posted by Ralph Goers <ra...@dslextreme.com>.
Yes, at least that way it shouldn’t impact the throughput of your application unless you are logging so much that whatever buffer or queueing it uses doesn’t fill up.

Ralph

> On Apr 3, 2018, at 9:30 PM, Ole Ersoy <ol...@gmail.com> wrote:
> 
> Just came across something related for amazon lambda:
> 
> https://medium.freecodecamp.org/how-to-implement-log-aggregation-for-aws-lambda-ca714bf02f48
> 
> Quote:
> 
> Dur­ing the exe­cu­tion of a Lamb­da func­tion, what­ev­er you write to std­out (for example, using|console.log|in Node.js) will be cap­tured by Lamb­da and sent to Cloud­Watch Logs asyn­chro­nous­ly in the back­ground.
> 
> Still does not answer the "Why" on the performance question, but I thought it was interesting ...
> 
> Cheers,
> 
> Ole
> 
> 
> 
> On 04/03/2018 04:25 PM, Ralph Goers wrote:
>> Thanks. That is helpful but it still doesn’t answer the question I was asking, although it does provide good documentation on what people are recommending for how to configure applications for the cloud.
>> 
>> Testing at Log4j has shown that writing to stdout is magnitudes slower than logging to a file, even when stdout is redirected to a file. What I am wondering, and still haven’t found an answer to, is whether this performance degradation is present when a Java app is running in a docker container and logs to stdout.
>> 
>> Ralph
>> 
>>> On Apr 3, 2018, at 11:44 AM, Ole Ersoy <ol...@gmail.com> wrote:
>>> 
>>> I accidentally deleted the original thread, but saw that there were some questions surrounding logging to stdout (I assume while running in a microservice dockerized environment).
>>> 
>>> You might find these article helpful:
>>> http://callistaenterprise.se/blogg/teknik/2017/07/29/building-microservices-part-7-distributed-tracing/
>>> http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
>>> 
>>> These cover log event collection for both ELK and Zipkin.  Parts 1-6 in the series are really good as well if you want to know more about microservices and security in general with the Spring and Netflix OSS stack.  Originally came across it while looking for OAuth material.
>>> 
>>> Cheers,
>>> Ole
>>> 
>> 
>> 
>

Re: Container Log Collection for Microservices

Posted by Ole Ersoy <ol...@gmail.com>.
Just came across something related for amazon lambda:

https://medium.freecodecamp.org/how-to-implement-log-aggregation-for-aws-lambda-ca714bf02f48

Quote:

Dur­ing the exe­cu­tion of a Lamb­da func­tion, what­ev­er you write to std­out (for example, using|console.log|in Node.js) will be cap­tured by Lamb­da and sent to Cloud­Watch Logs asyn­chro­nous­ly in the back­ground.

Still does not answer the "Why" on the performance question, but I thought it was interesting ...

Cheers,

Ole



On 04/03/2018 04:25 PM, Ralph Goers wrote:
> Thanks. That is helpful but it still doesn’t answer the question I was asking, although it does provide good documentation on what people are recommending for how to configure applications for the cloud.
>
> Testing at Log4j has shown that writing to stdout is magnitudes slower than logging to a file, even when stdout is redirected to a file. What I am wondering, and still haven’t found an answer to, is whether this performance degradation is present when a Java app is running in a docker container and logs to stdout.
>
> Ralph
>
>> On Apr 3, 2018, at 11:44 AM, Ole Ersoy <ol...@gmail.com> wrote:
>>
>> I accidentally deleted the original thread, but saw that there were some questions surrounding logging to stdout (I assume while running in a microservice dockerized environment).
>>
>> You might find these article helpful:
>> http://callistaenterprise.se/blogg/teknik/2017/07/29/building-microservices-part-7-distributed-tracing/
>> http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
>>
>> These cover log event collection for both ELK and Zipkin.  Parts 1-6 in the series are really good as well if you want to know more about microservices and security in general with the Spring and Netflix OSS stack.  Originally came across it while looking for OAuth material.
>>
>> Cheers,
>> Ole
>>
>
>

Re: Container Log Collection for Microservices

Posted by Ole Ersoy <ol...@gmail.com>.
I plan on playing with the microservices from the tutorial more in the near future.  I could:

1) Load test one microservice without stdout logging

2) Load test with stdout logging

3) See if there is any response degradation recorded by zipkin

Would be nice to see if Prometheus records any additional overhead as well (Next tutorial).

Once I get the setup I'll ping back.

IIUC the reason we are logging to stdout in docker containers is just so we can see the logging output when running the container during development.  We fire up the container and do some sanity testing with cURL and look at the log output with docker logs or docker compose logs.

Once we are happy we can turn off the logging to standard out and do file based logging or something more advanced with ELK and Zipkin.

Perhaps the docker layer handling the stdout stream is slow or that managing the buffer it uses is a performance bottle neck.

Cheers,
Ole




On 04/03/2018 04:25 PM, Ralph Goers wrote:
> Thanks. That is helpful but it still doesn’t answer the question I was asking, although it does provide good documentation on what people are recommending for how to configure applications for the cloud.
>
> Testing at Log4j has shown that writing to stdout is magnitudes slower than logging to a file, even when stdout is redirected to a file. What I am wondering, and still haven’t found an answer to, is whether this performance degradation is present when a Java app is running in a docker container and logs to stdout.
>
> Ralph
>
>> On Apr 3, 2018, at 11:44 AM, Ole Ersoy <ol...@gmail.com> wrote:
>>
>> I accidentally deleted the original thread, but saw that there were some questions surrounding logging to stdout (I assume while running in a microservice dockerized environment).
>>
>> You might find these article helpful:
>> http://callistaenterprise.se/blogg/teknik/2017/07/29/building-microservices-part-7-distributed-tracing/
>> http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
>>
>> These cover log event collection for both ELK and Zipkin.  Parts 1-6 in the series are really good as well if you want to know more about microservices and security in general with the Spring and Netflix OSS stack.  Originally came across it while looking for OAuth material.
>>
>> Cheers,
>> Ole
>>
>
>

Re: Container Log Collection for Microservices

Posted by Ralph Goers <ra...@dslextreme.com>.
Thanks. That is helpful but it still doesn’t answer the question I was asking, although it does provide good documentation on what people are recommending for how to configure applications for the cloud.  

Testing at Log4j has shown that writing to stdout is magnitudes slower than logging to a file, even when stdout is redirected to a file. What I am wondering, and still haven’t found an answer to, is whether this performance degradation is present when a Java app is running in a docker container and logs to stdout.

Ralph

> On Apr 3, 2018, at 11:44 AM, Ole Ersoy <ol...@gmail.com> wrote:
> 
> I accidentally deleted the original thread, but saw that there were some questions surrounding logging to stdout (I assume while running in a microservice dockerized environment).
> 
> You might find these article helpful:
> http://callistaenterprise.se/blogg/teknik/2017/07/29/building-microservices-part-7-distributed-tracing/
> http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
> 
> These cover log event collection for both ELK and Zipkin.  Parts 1-6 in the series are really good as well if you want to know more about microservices and security in general with the Spring and Netflix OSS stack.  Originally came across it while looking for OAuth material.
> 
> Cheers,
> Ole
>