You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mario Ds Briggs <ma...@in.ibm.com> on 2016/09/15 17:21:34 UTC
Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Hi,
I would like to use a Spark 2.0 History Server instance on spark1.6
generated eventlogs. (This is because clicking the refresh button in
browser, updates the UI with latest events, where-as in the 1.6 code base,
this does not happen)
My question is whether this is safe to do and are they any known issues
where event log contents of 1.6 are incompatible with a 2.0 history server.
Any official tracking of this ( i do know that the public SparkListener
interface didnt change across these versions)
thanks
Mario
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Parth Brahmbhatt <br...@gmail.com>.
The problem is we backported the Sql tab ui changes from 2.0 in our 1.6.1. They changed a parameter name in SQLMetricInfo. Still the community version is compatible, ours is not.
> On Sep 15, 2016, at 11:08 AM, Mario Ds Briggs <ma...@in.ibm.com> wrote:
>
> I had checked in 1.6.2 and it doesnt. I didnt check in lower versions. The history server logs do show a 'Replaying log path: file:xxx.inprogress' when the file is changed , but a refresh on UI doesnt show the new jobs/stages/tasks whatever
>
>
>
> thanks
> Mario
>
> <graycol.gif>Ryan Williams ---15/09/2016 11:19:56 pm---What is meant by: """
>
> From: Ryan Williams <ry...@gmail.com>
> To: Reynold Xin <rx...@databricks.com>, Mario Ds Briggs/India/IBM@IBMIN
> Cc: "dev@spark.apache.org" <de...@spark.apache.org>
> Date: 15/09/2016 11:19 pm
> Subject: Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
>
>
>
>
> What is meant by:
>
> """
> (This is because clicking the refresh button in browser, updates the UI with latest events, where-as in the 1.6 code base, this does not happen)
> """
>
> Hasn't refreshing the page updated all the information in the UI through the 1.x line?
>
>
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Kelvin Chu <2d...@gmail.com>.
We tried it and it works that v2.0 History Server can read the v1.6 logs.
Note that the UI has a regression. When there are too many jobs, the UI
will freeze because the new code tries to cache everything. We submitted a
JIRA: https://issues.apache.org/jira/browse/SPARK-17243
On Thu, Sep 15, 2016 at 11:08 AM, Mario Ds Briggs <ma...@in.ibm.com>
wrote:
> I had checked in 1.6.2 and it doesnt. I didnt check in lower versions. The
> history server logs do show a 'Replaying log path: file:xxx.inprogress'
> when the file is changed , but a refresh on UI doesnt show the new
> jobs/stages/tasks whatever
>
>
>
> thanks
> Mario
>
> [image: Inactive hide details for Ryan Williams ---15/09/2016 11:19:56
> pm---What is meant by: """]Ryan Williams ---15/09/2016 11:19:56 pm---What
> is meant by: """
>
> From: Ryan Williams <ry...@gmail.com>
> To: Reynold Xin <rx...@databricks.com>, Mario Ds Briggs/India/IBM@IBMIN
> Cc: "dev@spark.apache.org" <de...@spark.apache.org>
> Date: 15/09/2016 11:19 pm
> Subject: Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
> ------------------------------
>
>
>
> What is meant by:
>
> """
> (This is because clicking the refresh button in browser, updates the UI
> with latest events, where-as in the 1.6 code base, this does not happen)
> """
>
> Hasn't refreshing the page updated all the information in the UI through
> the 1.x line?
>
>
>
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Mario Ds Briggs <ma...@in.ibm.com>.
I had checked in 1.6.2 and it doesnt. I didnt check in lower versions. The
history server logs do show a 'Replaying log path: file:xxx.inprogress'
when the file is changed , but a refresh on UI doesnt show the new
jobs/stages/tasks whatever
thanks
Mario
From: Ryan Williams <ry...@gmail.com>
To: Reynold Xin <rx...@databricks.com>, Mario Ds
Briggs/India/IBM@IBMIN
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/09/2016 11:19 pm
Subject: Re: Compatibility of 1.6 spark.eventLog with a 2.0 History
Server
What is meant by:
"""
(This is because clicking the refresh button in browser, updates the UI
with latest events, where-as in the 1.6 code base, this does not happen)
"""
Hasn't refreshing the page updated all the information in the UI through
the 1.x line?
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Ryan Williams <ry...@gmail.com>.
What is meant by:
"""
(This is because clicking the refresh button in browser, updates the UI
with latest events, where-as in the 1.6 code base, this does not happen)
"""
Hasn't refreshing the page updated all the information in the UI through
the 1.x line?
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Reynold Xin <rx...@databricks.com>.
They should be compatible.
On Thu, Sep 15, 2016 at 10:21 AM, Mario Ds Briggs <ma...@in.ibm.com>
wrote:
> Hi,
>
> I would like to use a Spark 2.0 History Server instance on spark1.6
> generated eventlogs. (This is because clicking the refresh button in
> browser, updates the UI with latest events, where-as in the 1.6 code base,
> this does not happen)
>
> My question is whether this is safe to do and are they any known issues
> where event log contents of 1.6 are incompatible with a 2.0 history server.
> Any official tracking of this ( i do know that the public SparkListener
> interface didnt change across these versions)
>
>
> thanks
> Mario
>
Re: Compatibility of 1.6 spark.eventLog with a 2.0 History Server
Posted by Marcelo Vanzin <va...@cloudera.com>.
It should work fine. 2.0 dropped support for really old event logs
(pre-Spark 1.3 I think), but 1.6 should work, and if it doesn't it
should be considered a bug.
On Thu, Sep 15, 2016 at 10:21 AM, Mario Ds Briggs
<ma...@in.ibm.com> wrote:
> Hi,
>
> I would like to use a Spark 2.0 History Server instance on spark1.6
> generated eventlogs. (This is because clicking the refresh button in
> browser, updates the UI with latest events, where-as in the 1.6 code base,
> this does not happen)
>
> My question is whether this is safe to do and are they any known issues
> where event log contents of 1.6 are incompatible with a 2.0 history server.
> Any official tracking of this ( i do know that the public SparkListener
> interface didnt change across these versions)
>
>
> thanks
> Mario
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org