You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Attila Doroszlai <ad...@hortonworks.com> on 2017/08/21 16:23:34 UTC
Review Request 61786: AMBARI-21768. Spark History Server uses wrong
log dir
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/
-----------------------------------------------------------
Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21768
https://issues.apache.org/jira/browse/AMBARI-21768
Repository: ambari
Description
-------
* Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
* Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
Diffs
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
Diff: https://reviews.apache.org/r/61786/diff/1/
Testing
-------
Manual test according to steps in the bug.
Tested with both default and customized log directory location.
Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
Thanks,
Attila Doroszlai
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183346
-----------------------------------------------------------
Ship it!
No 4.2.5 chnaged needed?
- Sid Wagle
On Aug. 21, 2017, 4:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 4:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On Aug. 21, 2017, 10:59 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 380-381 (patched)
> > <https://reviews.apache.org/r/61786/diff/1/?file=1800806#file1800806line380>
> >
> > I thought that we went through properties like this and changed them to "hdp" ... Wouldn't this cause a problem with that replacement?
"Change stack references" happens before "Upgrade service configs", so the "iop" introduced here is not subject to replacement.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183388
-----------------------------------------------------------
On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 6:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183388
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 380-381 (patched)
<https://reviews.apache.org/r/61786/#comment259410>
I thought that we went through properties like this and changed them to "hdp" ... Wouldn't this cause a problem with that replacement?
- Jonathan Hurley
On Aug. 21, 2017, 12:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 12:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On Aug. 21, 2017, 7:09 p.m., Di Li wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 379 (patched)
> > <https://reviews.apache.org/r/61786/diff/1/?file=1800806#file1800806line379>
> >
> > Spark2 config in IOP 4.2.5 also has these properties. They are defined under different config types though, i.e spark2-default...
Upgrade from 4.2.5 works fine. IOP 4.2.5's default values for both `spark.eventLog.dir` and `spark.history.fs.logDirectory` match those of HDP (`hdfs:///spark2-history/`), thus the problem does not exist there.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183349
-----------------------------------------------------------
On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 6:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183349
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 379 (patched)
<https://reviews.apache.org/r/61786/#comment259382>
Spark2 config in IOP 4.2.5 also has these properties. They are defined under different config types though, i.e spark2-default...
- Di Li
On Aug. 21, 2017, 4:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 4:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On Aug. 21, 2017, 8:28 p.m., Bikas Saha wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 380 (patched)
> > <https://reviews.apache.org/r/61786/diff/1/?file=1800806#file1800806line380>
> >
> > This does not seem right because the log dir seems tied to a specific version. 4.2.0.0.
The version number does not matter, could be anything as long as it's the same as before the upgrade. This is the default value in IOP 4.2, and this replacement only applies for the non-customized case.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183361
-----------------------------------------------------------
On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 6:23 p.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61786: AMBARI-21768. Spark History Server uses
wrong log dir
Posted by Bikas Saha <bi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183361
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 380 (patched)
<https://reviews.apache.org/r/61786/#comment259391>
This does not seem right because the log dir seems tied to a specific version. 4.2.0.0.
- Bikas Saha
On Aug. 21, 2017, 9:23 a.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2017, 9:23 a.m.)
>
>
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they point to the cluster's existing Spark log directory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml ad20bf9d9427b6891c4d5944fa3b3e347dbffe12
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd
>
>
> Diff: https://reviews.apache.org/r/61786/diff/1/
>
>
> Testing
> -------
>
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History Server UI.
>
>
> Thanks,
>
> Attila Doroszlai
>
>