You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by canan chen <cc...@gmail.com> on 2015/08/07 10:20:42 UTC

Why use spark.history.fs.logDirectory instead of spark.eventLog.dir

Is there any reason that historyserver use another property for the event
log dir ? Thanks

Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir

Posted by canan chen <cc...@gmail.com>.
Thanks Andrew, make sense.

On Thu, Aug 20, 2015 at 4:40 AM, Andrew Or <an...@databricks.com> wrote:

> Hi Canan,
>
> The event log dir is a per-application setting whereas the history server
> is an independent service that serves history UIs from many applications.
> If you use history server locally then the `spark.history.fs.logDirectory`
> will happen to point to `spark.eventLog.dir`, but the use case it provides
> is broader than that.
>
> -Andrew
>
> 2015-08-19 5:13 GMT-07:00 canan chen <cc...@gmail.com>:
>
>> Anyone know about this ? Or do I miss something here ?
>>
>> On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
>>
>>> Is there any reason that historyserver use another property for the
>>> event log dir ? Thanks
>>>
>>
>>
>

Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir

Posted by Andrew Or <an...@databricks.com>.
Hi Canan,

The event log dir is a per-application setting whereas the history server
is an independent service that serves history UIs from many applications.
If you use history server locally then the `spark.history.fs.logDirectory`
will happen to point to `spark.eventLog.dir`, but the use case it provides
is broader than that.

-Andrew

2015-08-19 5:13 GMT-07:00 canan chen <cc...@gmail.com>:

> Anyone know about this ? Or do I miss something here ?
>
> On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
>
>> Is there any reason that historyserver use another property for the event
>> log dir ? Thanks
>>
>
>

Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir

Posted by canan chen <cc...@gmail.com>.
Anyone know about this ? Or do I miss something here ?

On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:

> Is there any reason that historyserver use another property for the event
> log dir ? Thanks
>

Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir

Posted by canan chen <cc...@gmail.com>.
Anyone know this ? Thanks

On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:

> Is there any reason that historyserver use another property for the event
> log dir ? Thanks
>