You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by canan chen <cc...@gmail.com> on 2015/08/07 10:20:42 UTC
Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Is there any reason that historyserver use another property for the event
log dir ? Thanks
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Posted by canan chen <cc...@gmail.com>.
Thanks Andrew, make sense.
On Thu, Aug 20, 2015 at 4:40 AM, Andrew Or <an...@databricks.com> wrote:
> Hi Canan,
>
> The event log dir is a per-application setting whereas the history server
> is an independent service that serves history UIs from many applications.
> If you use history server locally then the `spark.history.fs.logDirectory`
> will happen to point to `spark.eventLog.dir`, but the use case it provides
> is broader than that.
>
> -Andrew
>
> 2015-08-19 5:13 GMT-07:00 canan chen <cc...@gmail.com>:
>
>> Anyone know about this ? Or do I miss something here ?
>>
>> On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
>>
>>> Is there any reason that historyserver use another property for the
>>> event log dir ? Thanks
>>>
>>
>>
>
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Posted by Andrew Or <an...@databricks.com>.
Hi Canan,
The event log dir is a per-application setting whereas the history server
is an independent service that serves history UIs from many applications.
If you use history server locally then the `spark.history.fs.logDirectory`
will happen to point to `spark.eventLog.dir`, but the use case it provides
is broader than that.
-Andrew
2015-08-19 5:13 GMT-07:00 canan chen <cc...@gmail.com>:
> Anyone know about this ? Or do I miss something here ?
>
> On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
>
>> Is there any reason that historyserver use another property for the event
>> log dir ? Thanks
>>
>
>
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Posted by canan chen <cc...@gmail.com>.
Anyone know about this ? Or do I miss something here ?
On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
> Is there any reason that historyserver use another property for the event
> log dir ? Thanks
>
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Posted by canan chen <cc...@gmail.com>.
Anyone know this ? Thanks
On Fri, Aug 7, 2015 at 4:20 PM, canan chen <cc...@gmail.com> wrote:
> Is there any reason that historyserver use another property for the event
> log dir ? Thanks
>