You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Lian Jiang <ji...@gmail.com> on 2018/02/21 18:41:35 UTC
can log search support spark application logs?
Can I make my spark application log available for ambari log search so that
I can search them on log search UI? Or I have to setup another solr/elastic
search cluster for my application logs? Thanks for any hints.
Re: can log search support spark application logs?
Posted by Gonzalo Herreros <gh...@gmail.com>.
I think internally Ambari Search is really Solr, so you guess you could use
it directly
On 22 February 2018 at 22:19, Lian Jiang <ji...@gmail.com> wrote:
> Thanks. I want a log search UI like splunk/solr/kabana. It is interesting
> to see how to feed my application logs to log search.
>
> On Thu, Feb 22, 2018 at 2:02 PM, Gonzalo Herreros <gh...@gmail.com>
> wrote:
>
>> The logs are available through the Spark UI (which is the Application
>> Master link in YARN UI) on the Executors tab (stderr and stdout)
>> AFAIK, the log search can only index local files but you could have a
>> cronjob downloading then directly from hdfs.
>>
>> Gonzalo
>>
>>
>> On 22 February 2018 at 21:27, Lian Jiang <ji...@gmail.com> wrote:
>>
>>> Thanks for any help. What's the best way to get spark application logs?
>>> yarn command line is not ideal for spark streaming applications.
>>>
>>> On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com>
>>> wrote:
>>>
>>>> Can I make my spark application log available for ambari log search so
>>>> that I can search them on log search UI? Or I have to setup another
>>>> solr/elastic search cluster for my application logs? Thanks for any hints.
>>>>
>>>
>>>
>>
>
Re: can log search support spark application logs?
Posted by Don Bosco Durai <bo...@apache.org>.
Lian
LogSearch uses logfeeder reads the logs from the log file and pushes it to Ambari Infra Solr. If you know where your Sparks logs are configured to go, then you should be able to configure LogFeeder to send the logs to Ambari Infra Solr.
Bosco
From: Lian Jiang <ji...@gmail.com>
Reply-To: <us...@ambari.apache.org>
Date: Thursday, February 22, 2018 at 2:20 PM
To: <us...@ambari.apache.org>
Subject: Re: can log search support spark application logs?
Thanks. I want a log search UI like splunk/solr/kabana. It is interesting to see how to feed my application logs to log search.
On Thu, Feb 22, 2018 at 2:02 PM, Gonzalo Herreros <gh...@gmail.com> wrote:
The logs are available through the Spark UI (which is the Application Master link in YARN UI) on the Executors tab (stderr and stdout)
AFAIK, the log search can only index local files but you could have a cronjob downloading then directly from hdfs.
Gonzalo
On 22 February 2018 at 21:27, Lian Jiang <ji...@gmail.com> wrote:
Thanks for any help. What's the best way to get spark application logs? yarn command line is not ideal for spark streaming applications.
On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com> wrote:
Can I make my spark application log available for ambari log search so that I can search them on log search UI? Or I have to setup another solr/elastic search cluster for my application logs? Thanks for any hints.
Re: can log search support spark application logs?
Posted by Lian Jiang <ji...@gmail.com>.
Thanks. I want a log search UI like splunk/solr/kabana. It is interesting
to see how to feed my application logs to log search.
On Thu, Feb 22, 2018 at 2:02 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> The logs are available through the Spark UI (which is the Application
> Master link in YARN UI) on the Executors tab (stderr and stdout)
> AFAIK, the log search can only index local files but you could have a
> cronjob downloading then directly from hdfs.
>
> Gonzalo
>
>
> On 22 February 2018 at 21:27, Lian Jiang <ji...@gmail.com> wrote:
>
>> Thanks for any help. What's the best way to get spark application logs?
>> yarn command line is not ideal for spark streaming applications.
>>
>> On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com>
>> wrote:
>>
>>> Can I make my spark application log available for ambari log search so
>>> that I can search them on log search UI? Or I have to setup another
>>> solr/elastic search cluster for my application logs? Thanks for any hints.
>>>
>>
>>
>
Re: can log search support spark application logs?
Posted by Gonzalo Herreros <gh...@gmail.com>.
The logs are available through the Spark UI (which is the Application
Master link in YARN UI) on the Executors tab (stderr and stdout)
AFAIK, the log search can only index local files but you could have a
cronjob downloading then directly from hdfs.
Gonzalo
On 22 February 2018 at 21:27, Lian Jiang <ji...@gmail.com> wrote:
> Thanks for any help. What's the best way to get spark application logs?
> yarn command line is not ideal for spark streaming applications.
>
> On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com>
> wrote:
>
>> Can I make my spark application log available for ambari log search so
>> that I can search them on log search UI? Or I have to setup another
>> solr/elastic search cluster for my application logs? Thanks for any hints.
>>
>
>
Re: can log search support spark application logs?
Posted by Lian Jiang <ji...@gmail.com>.
Thanks for any help. What's the best way to get spark application logs?
yarn command line is not ideal for spark streaming applications.
On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <ji...@gmail.com> wrote:
> Can I make my spark application log available for ambari log search so
> that I can search them on log search UI? Or I have to setup another
> solr/elastic search cluster for my application logs? Thanks for any hints.
>