You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2017/07/15 00:14:40 UTC
Review Request 60886: AMBARI-21481. Upgrading IOP cluster with Spark2
to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/
-----------------------------------------------------------
Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-21481
https://issues.apache.org/jira/browse/AMBARI-21481
Repository: ambari
Description
-------
STR:
* Install BigInsights 4.2.5 with Spark2
* Upgrade to Ambari 2.5.2
* Start Ambari, which will fail in the DB consistency check
```
2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
```
Looks like this config is never selected,
```
ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
JOIN clusterconfig c ON m.config_id = c.config_id
JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
ORDER BY sc.version, c.type_name;
service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
--------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
(2 rows)
```
This is because the config type has a single config with an empty value, so perhaps UI never selects it.
```
<configuration supports_final="true">
<property>
<name>content</name>
<description>Spark2-javaopts-properties</description>
<value> </value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
```
Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
The only config type to hit this issue is spark2-javaopts-properties
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
Diff: https://reviews.apache.org/r/60886/diff/1/
Testing
-------
Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
Thanks,
Alejandro Fernandez
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On July 15, 2017, 12:40 a.m., Sid Wagle wrote:
> > What happens to configmappings unselected due to service delete? What wouldbe side effects of selecting those?
Before a service is deleted, it should have exactly one config mapping for a certain config type as selected (and in this case that condition is not met). If a service is deleted, I need to check if we unselect all of its config mappings; but either way it shouldn't matter because if the service is re-added we always generate a new config version.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180617
-----------------------------------------------------------
On July 15, 2017, 12:20 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 15, 2017, 12:20 a.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/2/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180617
-----------------------------------------------------------
What happens to configmappings unselected due to service delete? What wouldbe side effects of selecting those?
- Sid Wagle
On July 15, 2017, 12:20 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 15, 2017, 12:20 a.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/2/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On July 17, 2017, 3:29 p.m., Jonathan Hurley wrote:
> > So, this is an env file with no content? Doesn't that mean that it's optional and doesn't need to be created? Can we simply use the <optional> tag (see AMBARI-21364) and have the DB checker skip it?
The config is already saved to the DB consistency checker applies its business logic that at least one config mapping must be selected. If this were HDP, I could also mark it as <optional> but since it's for the BigInsights stack it won't affect any new clusters.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180681
-----------------------------------------------------------
On July 15, 2017, 12:20 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 15, 2017, 12:20 a.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/2/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180681
-----------------------------------------------------------
So, this is an env file with no content? Doesn't that mean that it's optional and doesn't need to be created? Can we simply use the <optional> tag (see AMBARI-21364) and have the DB checker skip it?
- Jonathan Hurley
On July 14, 2017, 8:20 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 14, 2017, 8:20 p.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/2/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/
-----------------------------------------------------------
(Updated July 17, 2017, 9:26 p.m.)
Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-21481
https://issues.apache.org/jira/browse/AMBARI-21481
Repository: ambari
Description
-------
STR:
* Install BigInsights 4.2.5 with Spark2
* Upgrade to Ambari 2.5.2
* Start Ambari, which will fail in the DB consistency check
```
2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
```
Looks like this config is never selected,
```
ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
JOIN clusterconfig c ON m.config_id = c.config_id
JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
ORDER BY sc.version, c.type_name;
service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
--------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
(2 rows)
```
This is because the config type has a single config with an empty value, so perhaps UI never selects it.
```
<configuration supports_final="true">
<property>
<name>content</name>
<description>Spark2-javaopts-properties</description>
<value> </value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
```
Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
The only config type to hit this issue is spark2-javaopts-properties
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
Diff: https://reviews.apache.org/r/60886/diff/5/
Changes: https://reviews.apache.org/r/60886/diff/4-5/
Testing
-------
Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
Thanks,
Alejandro Fernandez
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180724
-----------------------------------------------------------
Fix it, then Ship it!
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 218-224 (patched)
<https://reviews.apache.org/r/60886/#comment255941>
Can we change this so that it doesn't mention stack upgrades or HDP/BI? The problem is on Ambari upgrade, right? And it could happen for any config?
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 228 (patched)
<https://reviews.apache.org/r/60886/#comment255942>
Can we do away with this check? I don't think we'd be upgrading without any clusters :)
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 230 (patched)
<https://reviews.apache.org/r/60886/#comment255943>
This should probably log the types we're enforcing now, which seems to be only 1.
- Jonathan Hurley
On July 17, 2017, 3:15 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 3:15 p.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/4/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/
-----------------------------------------------------------
(Updated July 17, 2017, 7:15 p.m.)
Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-21481
https://issues.apache.org/jira/browse/AMBARI-21481
Repository: ambari
Description
-------
STR:
* Install BigInsights 4.2.5 with Spark2
* Upgrade to Ambari 2.5.2
* Start Ambari, which will fail in the DB consistency check
```
2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
```
Looks like this config is never selected,
```
ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
JOIN clusterconfig c ON m.config_id = c.config_id
JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
ORDER BY sc.version, c.type_name;
service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
--------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
(2 rows)
```
This is because the config type has a single config with an empty value, so perhaps UI never selects it.
```
<configuration supports_final="true">
<property>
<name>content</name>
<description>Spark2-javaopts-properties</description>
<value> </value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
```
Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
The only config type to hit this issue is spark2-javaopts-properties
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
Diff: https://reviews.apache.org/r/60886/diff/4/
Changes: https://reviews.apache.org/r/60886/diff/3-4/
Testing
-------
Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
Thanks,
Alejandro Fernandez
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/
-----------------------------------------------------------
(Updated July 17, 2017, 7:12 p.m.)
Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-21481
https://issues.apache.org/jira/browse/AMBARI-21481
Repository: ambari
Description
-------
STR:
* Install BigInsights 4.2.5 with Spark2
* Upgrade to Ambari 2.5.2
* Start Ambari, which will fail in the DB consistency check
```
2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
```
Looks like this config is never selected,
```
ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
JOIN clusterconfig c ON m.config_id = c.config_id
JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
ORDER BY sc.version, c.type_name;
service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
--------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
(2 rows)
```
This is because the config type has a single config with an empty value, so perhaps UI never selects it.
```
<configuration supports_final="true">
<property>
<name>content</name>
<description>Spark2-javaopts-properties</description>
<value> </value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
```
Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
The only config type to hit this issue is spark2-javaopts-properties
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
Diff: https://reviews.apache.org/r/60886/diff/3/
Changes: https://reviews.apache.org/r/60886/diff/2-3/
Testing
-------
Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
Thanks,
Alejandro Fernandez
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On July 17, 2017, 6:26 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml
> > Line 82 (original), 82 (patched)
> > <https://reviews.apache.org/r/60886/diff/2/?file=1777210#file1777210line82>
> >
> > Why indented?
Fixed
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180710
-----------------------------------------------------------
On July 17, 2017, 7:15 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 7:15 p.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/4/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On July 17, 2017, 6:26 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
> > Lines 214-221 (patched)
> > <https://reviews.apache.org/r/60886/diff/2/?file=1777207#file1777207line214>
> >
> > I don't think this comment is correct - this doesn't deal with a stack upgrade - it deals with upgrading Ambari, right?
> >
> > Also - what happens with services which were deleted. Their configurations remain unselected, no?
This issue is due to Ambari Upgrade, which fails starting the server due to the DB consistency checks we have in Ambari 2.5.2
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180710
-----------------------------------------------------------
On July 17, 2017, 7:12 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 7:12 p.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 3c8686c
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/3/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180710
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 214-221 (patched)
<https://reviews.apache.org/r/60886/#comment255933>
I don't think this comment is correct - this doesn't deal with a stack upgrade - it deals with upgrading Ambari, right?
Also - what happens with services which were deleted. Their configurations remain unselected, no?
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml
Line 82 (original), 82 (patched)
<https://reviews.apache.org/r/60886/#comment255932>
Why indented?
- Jonathan Hurley
On July 14, 2017, 8:20 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 14, 2017, 8:20 p.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/2/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/
-----------------------------------------------------------
(Updated July 15, 2017, 12:20 a.m.)
Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
Changes
-------
Added java doc
Bugs: AMBARI-21481
https://issues.apache.org/jira/browse/AMBARI-21481
Repository: ambari
Description
-------
STR:
* Install BigInsights 4.2.5 with Spark2
* Upgrade to Ambari 2.5.2
* Start Ambari, which will fail in the DB consistency check
```
2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
```
Looks like this config is never selected,
```
ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
JOIN clusterconfig c ON m.config_id = c.config_id
JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
ORDER BY sc.version, c.type_name;
service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
--------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
(2 rows)
```
This is because the config type has a single config with an empty value, so perhaps UI never selects it.
```
<configuration supports_final="true">
<property>
<name>content</name>
<description>Spark2-javaopts-properties</description>
<value> </value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
```
Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
The only config type to hit this issue is spark2-javaopts-properties
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
Diff: https://reviews.apache.org/r/60886/diff/2/
Changes: https://reviews.apache.org/r/60886/diff/1-2/
Testing
-------
Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
Thanks,
Alejandro Fernandez
Re: Review Request 60886: AMBARI-21481. Upgrading IOP cluster with
Spark2 to Ambari 2.5.2 fails on start because config mapping
spark2-javaopts-properties is never selected
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60886/#review180614
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 214 (patched)
<https://reviews.apache.org/r/60886/#comment255815>
Will add javadoc.
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 261 (patched)
<https://reviews.apache.org/r/60886/#comment255818>
The only config type that had this issue was spark2-javaopts-properties so I feel confident keeping it generic.
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml
Lines 27 (patched)
<https://reviews.apache.org/r/60886/#comment255816>
Will raise a Jira for Ambari 2.5.3 for the UI to select config types that only contain configs with empty values.
- Alejandro Fernandez
On July 15, 2017, 12:14 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60886/
> -----------------------------------------------------------
>
> (Updated July 15, 2017, 12:14 a.m.)
>
>
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-21481
> https://issues.apache.org/jira/browse/AMBARI-21481
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
>
> * Install BigInsights 4.2.5 with Spark2
> * Upgrade to Ambari 2.5.2
> * Start Ambari, which will fail in the DB consistency check
>
> ```
> 2017-07-07 01:07:01,881 ERROR - You have non selected configs: spark2-javaopts-properties for service SPARK2 from cluster c1!
> ```
>
> Looks like this config is never selected,
>
> ```
> ambari=> SELECT sc.service_name, sc.version, sc.user_name, TO_TIMESTAMP(sc.create_timestamp/1000), sc.stack_id, sc.note,
> c.version AS type_version, c.version_tag, c.type_name, c.stack_id, TO_TIMESTAMP(c.create_timestamp/1000) AS clusterconfig_created,
> TO_TIMESTAMP(ccm.create_timestamp/1000) AS clusterconfigmapping_created, ccm.selected, ccm.user_name
> FROM serviceconfig sc JOIN serviceconfigmapping m ON sc.service_config_id = m.service_config_id
> JOIN clusterconfig c ON m.config_id = c.config_id
> JOIN clusterconfigmapping ccm ON c.type_name = ccm.type_name AND c.version_tag = ccm.version_tag
> WHERE sc.service_name = 'SPARK2' and c.type_name = 'spark2-javaopts-properties'
> ORDER BY sc.version, c.type_name;
> service_name | version | user_name | to_timestamp | stack_id | note | type_version | version_tag | type_name | stack_id | clusterconfig_created | clusterconfigmapping_created | selected | user_name
> --------------+---------+----------------+------------------------+----------+-----------------------------------+--------------+-------------+----------------------------+----------+------------------------+------------------------------+----------+-----------
> SPARK2 | 1 | admin | 2017-07-11 20:17:01+00 | 3 | Initial configurations for Spark2 | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> SPARK2 | 2 | ambari-upgrade | 2017-07-13 20:33:50+00 | 3 | | 1 | version1 | spark2-javaopts-properties | 3 | 2017-07-11 20:17:00+00 | 2017-07-11 20:17:01+00 | 0 | admin
> (2 rows)
> ```
>
> This is because the config type has a single config with an empty value, so perhaps UI never selects it.
> ```
> <configuration supports_final="true">
> <property>
> <name>content</name>
> <description>Spark2-javaopts-properties</description>
> <value> </value>
> <on-ambari-upgrade add="true"/>
> </property>
> </configuration>
> ```
>
> Fix will be for Ambari 2.5.2 upgrade to select exactly once config for each config type if none are selected.
> The only config type to hit this issue is spark2-javaopts-properties
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java 6d2ab84
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/configuration/spark-javaopts-properties.xml a197e34
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/configuration/spark2-javaopts-properties.xml f8d50fc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SPARK2/metainfo.xml bf75f47
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/configuration/spark-javaopts-properties.xml 77a7282
>
>
> Diff: https://reviews.apache.org/r/60886/diff/1/
>
>
> Testing
> -------
>
> Verified on local cluster. Unit test for UpgradeCatalog252Test.java passed.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>