Upgrading spark history server, no logs showing.

[bbarks <br...@pm.me>]

Hi, 

We have multiple installations of spark installed on our clusters. They
reside in different directories which the jobs point to when they run.

For a couple of years now, we've run our history server off spark 2.0.2. We
have 2.1.2, 2.2.1 and 2.3.0 installed as well. I've tried upgrading to run
the server out of the 2.3.0 install. The UI loads, but will not show logs.

For fun, I then tried 2.2.1, same deal. However, when I ran 2.1.2, it works
(albeit with a JS error about missing data in some table cell or row).

Are there any special steps to upgrading the history server between spark
versions? I've combed over settings multiple times, it all seems fine.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Upgrading spark history server, no logs showing.

[bbarks <br...@pm.me>]

I finally circled back and tinkered with this, eventually finding the
solution. It turned out to be HDFS permissions on the history files.

For whatever reason, our HDFS perms worked fine with spark 2.0.2 and 2.1.2,
but when we ran spark 2.3.0 it wouldn't load any history in the UI.

I found out the history files are written with perms 0770, which is
hard-coded in the spark source. I just had to chown the history directory to
the same group the history service user is in, then set the facl so the
group has default permissions on all files added to the directory.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org