Re: can log search support spark application logs?

[Don Bosco Durai <bo...@apache.org>]

Lian

 

LogSearch uses logfeeder reads the logs from the log file and pushes it to Ambari Infra Solr. If you know where your Sparks logs are configured to go, then you should be able to configure LogFeeder to send the logs to Ambari Infra Solr.

 

Bosco

 

 

From: Lian Jiang <ji...@gmail.com>
Reply-To: <us...@ambari.apache.org>
Date: Thursday, February 22, 2018 at 2:20 PM
To: <us...@ambari.apache.org>
Subject: Re: can log search support spark application logs?

 

Thanks. I want a log search UI like splunk/solr/kabana. It is interesting to see how to feed my application logs to log search.

 

On Thu, Feb 22, 2018 at 2:02 PM, Gonzalo Herreros <gh...@gmail.com> wrote:

The logs are available through the Spark UI (which is the Application Master link in YARN UI) on the Executors tab (stderr and stdout)
AFAIK, the log search can only index local files but you could have a cronjob downloading then directly from hdfs.

 

Gonzalo

 

 

On 22 February 2018 at 21:27, Lian Jiang <ji...@gmail.com> wrote:

Thanks for any help. What's the best way to get spark application logs? yarn command line is not ideal for spark streaming applications.

 

On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com> wrote:

Can I make my spark application log available for ambari log search so that I can search them on log search UI? Or I have to setup another solr/elastic search cluster for my application logs? Thanks for any hints.