You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Attila Doroszlai <ad...@hortonworks.com> on 2017/07/27 14:28:08 UTC
Review Request 61178: AMBARI-21572. Spark configuration mismatch post
express upgrade
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
Bugs: AMBARI-21572
https://issues.apache.org/jira/browse/AMBARI-21572
Repository: ambari
Description
-------
Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
Diffs
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
Diff: https://reviews.apache.org/r/61178/diff/1/
Testing
-------
Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
Thanks,
Attila Doroszlai
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On July 27, 2017, 4:35 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 268 (patched)
> > <https://reviews.apache.org/r/61178/diff/1/?file=1784488#file1784488line268>
> >
> > Can it be deleted unconditionally or only if the value is exactly the hdfs location? I suspect if the cluster uses some other DFS.
The current patch deletes it only if it matches the exact location. It can be changed to unconditional, if desired.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181565
-----------------------------------------------------------
On July 27, 2017, 4:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 4:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181565
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
Lines 268 (patched)
<https://reviews.apache.org/r/61178/#comment257158>
Can it be deleted unconditionally or only if the value is exactly the hdfs location? I suspect if the cluster uses some other DFS.
- Sumit Mohanty
On July 27, 2017, 2:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 2:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On July 27, 2017, 9:43 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 268 (patched)
> > <https://reviews.apache.org/r/61178/diff/1/?file=1784488#file1784488line268>
> >
> > Is this deleting the config with the original value from the BigInsights stack?
> > Would EU to HDP 2.6 create it with a new config value?
The value is already changed by the previous "update stack references" step, hence `hdp` in it.
`spark.yarn.archive` is not added during EU nor used by HDP.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181604
-----------------------------------------------------------
On July 27, 2017, 4:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 4:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On July 27, 2017, 7:43 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 268 (patched)
> > <https://reviews.apache.org/r/61178/diff/1/?file=1784488#file1784488line268>
> >
> > Is this deleting the config with the original value from the BigInsights stack?
> > Would EU to HDP 2.6 create it with a new config value?
>
> Attila Doroszlai wrote:
> The value is already changed by the previous "update stack references" step, hence `hdp` in it.
> `spark.yarn.archive` is not added during EU nor used by HDP.
Since not used, then delete the config without checking the value
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181604
-----------------------------------------------------------
On July 27, 2017, 2:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 2:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181604
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
Lines 268 (patched)
<https://reviews.apache.org/r/61178/#comment257191>
Is this deleting the config with the original value from the BigInsights stack?
Would EU to HDP 2.6 create it with a new config value?
- Alejandro Fernandez
On July 27, 2017, 2:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 2:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181738
-----------------------------------------------------------
Ship it!
Ship It!
- Di Li
On July 28, 2017, 9:44 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 28, 2017, 9:44 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` and `spark.thriftserver.ui.port` properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml a7fdeecd86533c70975bdc072815b3b7d5abed2b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/3/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Properties are removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/
-----------------------------------------------------------
(Updated July 28, 2017, 11:44 p.m.)
Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
Changes
-------
Also remove spark.thriftserver.ui.port, no longer used
Bugs: AMBARI-21572
https://issues.apache.org/jira/browse/AMBARI-21572
Repository: ambari
Description (updated)
-------
Remove unnecessary `spark.yarn.archive` and `spark.thriftserver.ui.port` properties
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml a7fdeecd86533c70975bdc072815b3b7d5abed2b
Diff: https://reviews.apache.org/r/61178/diff/3/
Changes: https://reviews.apache.org/r/61178/diff/2-3/
Testing (updated)
-------
Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Properties are removed during upgrade and no longer present in Spark configs.
Thanks,
Attila Doroszlai
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On July 28, 2017, 8:17 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 266-269 (patched)
> > <https://reviews.apache.org/r/61178/diff/2/?file=1785094#file1785094line266>
> >
> > This just deletes the property outright. The description of the bug says that if it's been converted to leave it. Is that actually the desired behavior?
"If this property isn't used by HDP it should be deleted during the stack upgrade." -- this is what's implemented. Also see comment above by Alejandro: "Since not used, then delete the config without checking the value".
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181696
-----------------------------------------------------------
On July 28, 2017, 9:22 a.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 28, 2017, 9:22 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/2/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181696
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
Lines 266-269 (patched)
<https://reviews.apache.org/r/61178/#comment257329>
This just deletes the property outright. The description of the bug says that if it's been converted to leave it. Is that actually the desired behavior?
- Jonathan Hurley
On July 28, 2017, 3:22 a.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 28, 2017, 3:22 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/2/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/
-----------------------------------------------------------
(Updated July 28, 2017, 9:22 a.m.)
Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
Changes
-------
unconditionally delete spark.yarn.archive
Bugs: AMBARI-21572
https://issues.apache.org/jira/browse/AMBARI-21572
Repository: ambari
Description
-------
Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
Diff: https://reviews.apache.org/r/61178/diff/2/
Changes: https://reviews.apache.org/r/61178/diff/1-2/
Testing
-------
Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
Thanks,
Attila Doroszlai
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On July 27, 2017, 4:38 p.m., Di Li wrote:
> > Hello Attila,
> >
> > The issue with the alert (2nd part of the issue), was BI stack has its own sparks thrift server alert that check hardcoded port 4038. HDP stack uses a different alert for the same purpose.
> >
> > This manifests on the UI as "two" Spark Thrift Server alert that one with status OK (HDP one, good, should be kept, alert id spark2_thriftserver_status"" ), one with Critical (IOP one alert id "SPARK2_THRIFTSERVER_PROCESS", wrong, should be removed, or the port updated from 4038 to 4040)
>
> Attila Doroszlai wrote:
> Hi Di, the alert problem will be addressed in a separate Jira, because it may be a less straightforward change.
>
> Di Li wrote:
> Got it, could you please subscribe me on that JIRA ? thank you very much.
Sure, once it's created.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181566
-----------------------------------------------------------
On July 27, 2017, 4:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 4:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Attila Doroszlai <ad...@hortonworks.com>.
> On July 27, 2017, 4:38 p.m., Di Li wrote:
> > Hello Attila,
> >
> > The issue with the alert (2nd part of the issue), was BI stack has its own sparks thrift server alert that check hardcoded port 4038. HDP stack uses a different alert for the same purpose.
> >
> > This manifests on the UI as "two" Spark Thrift Server alert that one with status OK (HDP one, good, should be kept, alert id spark2_thriftserver_status"" ), one with Critical (IOP one alert id "SPARK2_THRIFTSERVER_PROCESS", wrong, should be removed, or the port updated from 4038 to 4040)
Hi Di, the alert problem will be addressed in a separate Jira, because it may be a less straightforward change.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181566
-----------------------------------------------------------
On July 27, 2017, 4:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 4:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Di Li <di...@ca.ibm.com>.
> On July 27, 2017, 2:38 p.m., Di Li wrote:
> > Hello Attila,
> >
> > The issue with the alert (2nd part of the issue), was BI stack has its own sparks thrift server alert that check hardcoded port 4038. HDP stack uses a different alert for the same purpose.
> >
> > This manifests on the UI as "two" Spark Thrift Server alert that one with status OK (HDP one, good, should be kept, alert id spark2_thriftserver_status"" ), one with Critical (IOP one alert id "SPARK2_THRIFTSERVER_PROCESS", wrong, should be removed, or the port updated from 4038 to 4040)
>
> Attila Doroszlai wrote:
> Hi Di, the alert problem will be addressed in a separate Jira, because it may be a less straightforward change.
Got it, could you please subscribe me on that JIRA ? thank you very much.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181566
-----------------------------------------------------------
On July 27, 2017, 2:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 2:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>
Re: Review Request 61178: AMBARI-21572. Spark configuration mismatch
post express upgrade
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61178/#review181566
-----------------------------------------------------------
Hello Attila,
The issue with the alert (2nd part of the issue), was BI stack has its own sparks thrift server alert that check hardcoded port 4038. HDP stack uses a different alert for the same purpose.
This manifests on the UI as "two" Spark Thrift Server alert that one with status OK (HDP one, good, should be kept, alert id spark2_thriftserver_status"" ), one with Critical (IOP one alert id "SPARK2_THRIFTSERVER_PROCESS", wrong, should be removed, or the port updated from 4038 to 4040)
- Di Li
On July 27, 2017, 2:28 p.m., Attila Doroszlai wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61178/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 2:28 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Eric Yang, Jonathan Hurley, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-21572
> https://issues.apache.org/jira/browse/AMBARI-21572
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Remove unnecessary `spark.yarn.archive` property (if value matches IOP default one, except it's already converted to `hdfs:///hdp/...`).
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml f71e91adcb1091074f83f1f09759968de50c82a9
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml 9d3cc1e48e090f8a1f43cfaeb024dbeabdf4384b
>
>
> Diff: https://reviews.apache.org/r/61178/diff/1/
>
>
> Testing
> -------
>
> Tested upgrade from BI 4.2.5 to HDP 2.6 manually. Property is removed during upgrade and no longer present in Spark configs.
>
>
> Thanks,
>
> Attila Doroszlai
>
>