You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Cheng Pan <ch...@apache.org> on 2023/03/14 12:44:43 UTC

Re: spark on k8s daemonset collect log

The filebeat supports multiline matching, here is an example[1]

BTW, I’m working on External Log Service integration[2], it may be useful
in your case, feel free to review/left comments

[1]
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline
[2] https://github.com/apache/spark/pull/38357

Thanks,
Cheng Pan


On Mar 14, 2023 at 16:36:45, 404 <yx...@126.com> wrote:

> hi, all
>
> Spark runs on k8s, uses daemonset filebeat to collect logs, and writes
> them to elasticsearch. The docker logs are in json format, and each line is
> a json string. How to merge multi-line exceptions?
>
>