Camel Tracer and Logstash

[Karts <ka...@gmail.com>]

Has anyone managed to integrate Camel logs with Logstash/Elasticsearch?

I've got tracer and MDC logging enabled in my CamelContext and am sending
logs to Logstash via log4j TCP appender. This is working great (especially
MDC fields), but I'm trying to further parse and filter the message to
individual fields in Elasticsearch.

For example, my message is:
ID-dx123abc001-abc-com-42264-1475782902034-0-193168 >>> (system-sql)
setHeader[Content-Type, application/json] -->
marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55] <<<
Pattern:InOut, Headers:{Accept-Encoding=gzip, deflate, Authorization=Basic
aaaaXRpeDpreG1pZGFz,
breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169,
CamelHttpMethod=GET, CamelHttpPath=, CamelHttpQuery=name=new-strategies,
CamelHttpServletRequest=Request(GET
/esb/system/sql?name=new-strategies)@6aaa8b00,
CamelHttpServletResponse=HTTP/1.1 200 , CamelHttpUri=/esb/system/sql,
CamelHttpUrl=http://system.abc.com:51950/esb/system/sql,
CamelJdbcColumnNames=[STRATEGY_NAME], CamelJdbcRowCount=2,
CamelServletContextPath=/esb/system/sql, Connection=keep-alive,
Content-Type=application/json, Host=system.abc.com:51950,
name=new-strategies, User-Agent=Apache-HttpClient/4.2.1 (java 1.5)},
BodyType:java.util.ArrayList, Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2
}]

And I would like logstash to parse and filter this so that it is stored in
Elasticsearch as individual fields:
...
from=setHeader[Content-Type, application/json]
to=marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55] 
breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169
CamelJdbcColumnNames=[STRATEGY_NAME]
CamelJdbcRowCount=2
BodyType:java.util.ArrayList
Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2 }]
etc.

Has anyone been able to do this? I'm struggling with getting this to work
with grok or ruby in the logstash config. Any tips are appreciated!

Thanks



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Tracer-and-Logstash-tp5788502.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel Tracer and Logstash

[Karts <ka...@gmail.com>]

In case anyone needs it, we used grok to achieve this:

if [path] =~ "Tracer" {
        grok {
                patterns_dir => ["./extra_patterns"]
                match => {
                        "message" => "^\n%{USERNAME:id} >>>
\(%{USERNAME:routeId}\) %{GREEDYDATA:source} --> %{GREEDYDATA:destination}
<<<\n, Pattern:%{WORD:pattern}\n,
Headers:{%{MULTILINEGREEDYDATA:headers}}\n, BodyType:%{JAVAFILE:bodyType}\n,
Body:%{GREEDYDATA:body}"
                }
        }
        kv {
                source => "headers"
                field_split => ","
        }
  }




--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Tracer-and-Logstash-tp5788502p5788689.html
Sent from the Camel - Users mailing list archive at Nabble.com.