You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2017/04/27 16:16:29 UTC
Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-20875
https://issues.apache.org/jira/browse/AMBARI-20875
Repository: ambari
Description
-------
When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
{noformat}
2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes-s
ite-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
{noformat}
These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
A few options:
- When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
- Make the warning smarter somehow to ignore these since it's expected they are orphaned.
-- Somehow determine the service which should own the config and see if it exists in the cluster?
-- Add a new column to {{clusterconfig}} to mark it as deleted?
To clean these warnings, we had to:
{code}
CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
(SELECT
cc.config_id,
cc.type_name,
cc.version_tag
FROM clusterconfig cc, clusterconfigmapping ccm
WHERE cc.config_id NOT IN (SELECT
scm.config_id
FROM serviceconfigmapping scm)
AND cc.type_name != 'cluster-env'
AND cc.type_name = ccm.type_name
AND cc.version_tag = ccm.version_tag);
DELETE FROM clusterconfigmapping
WHERE EXISTS
(SELECT 1 FROM orphaned_configs
WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
SELECT * FROM orphaned_configs;
DROP TABLE orphaned_configs;
{code}
I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
Diff: https://reviews.apache.org/r/58802/diff/1/
Testing
-------
mvn clean test
Also manual check on Derby
Thanks,
Dmitro Lisnichenko
Re: Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/#review173265
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
Lines 563 (patched)
<https://reviews.apache.org/r/58802/#comment246303>
Use ==
- Alejandro Fernandez
On April 27, 2017, 4:16 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58802/
> -----------------------------------------------------------
>
> (Updated April 27, 2017, 4:16 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-20875
> https://issues.apache.org/jira/browse/AMBARI-20875
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
>
> After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
>
> {noformat}
> 2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes
-site-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
> {noformat}
>
> These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
>
> A few options:
> - When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
>
> - Make the warning smarter somehow to ignore these since it's expected they are orphaned.
> -- Somehow determine the service which should own the config and see if it exists in the cluster?
> -- Add a new column to {{clusterconfig}} to mark it as deleted?
>
>
> To clean these warnings, we had to:
> {code}
> CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
> (SELECT
> cc.config_id,
> cc.type_name,
> cc.version_tag
> FROM clusterconfig cc, clusterconfigmapping ccm
> WHERE cc.config_id NOT IN (SELECT
> scm.config_id
> FROM serviceconfigmapping scm)
> AND cc.type_name != 'cluster-env'
> AND cc.type_name = ccm.type_name
> AND cc.version_tag = ccm.version_tag);
>
> DELETE FROM clusterconfigmapping
> WHERE EXISTS
> (SELECT 1 FROM orphaned_configs
> WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
>
> DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
>
> SELECT * FROM orphaned_configs;
>
> DROP TABLE orphaned_configs;
> {code}
>
> I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
> So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58802/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
> Also manual check on Derby
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/#review173264
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On April 27, 2017, 4:16 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58802/
> -----------------------------------------------------------
>
> (Updated April 27, 2017, 4:16 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-20875
> https://issues.apache.org/jira/browse/AMBARI-20875
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
>
> After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
>
> {noformat}
> 2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes
-site-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
> {noformat}
>
> These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
>
> A few options:
> - When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
>
> - Make the warning smarter somehow to ignore these since it's expected they are orphaned.
> -- Somehow determine the service which should own the config and see if it exists in the cluster?
> -- Add a new column to {{clusterconfig}} to mark it as deleted?
>
>
> To clean these warnings, we had to:
> {code}
> CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
> (SELECT
> cc.config_id,
> cc.type_name,
> cc.version_tag
> FROM clusterconfig cc, clusterconfigmapping ccm
> WHERE cc.config_id NOT IN (SELECT
> scm.config_id
> FROM serviceconfigmapping scm)
> AND cc.type_name != 'cluster-env'
> AND cc.type_name = ccm.type_name
> AND cc.version_tag = ccm.version_tag);
>
> DELETE FROM clusterconfigmapping
> WHERE EXISTS
> (SELECT 1 FROM orphaned_configs
> WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
>
> DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
>
> SELECT * FROM orphaned_configs;
>
> DROP TABLE orphaned_configs;
> {code}
>
> I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
> So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58802/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
> Also manual check on Derby
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/
-----------------------------------------------------------
(Updated May 3, 2017, 1:59 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
-------
fixed review comments
Bugs: AMBARI-20875
https://issues.apache.org/jira/browse/AMBARI-20875
Repository: ambari
Description
-------
When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
{noformat}
2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes-s
ite-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
{noformat}
These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
A few options:
- When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
- Make the warning smarter somehow to ignore these since it's expected they are orphaned.
-- Somehow determine the service which should own the config and see if it exists in the cluster?
-- Add a new column to {{clusterconfig}} to mark it as deleted?
To clean these warnings, we had to:
{code}
CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
(SELECT
cc.config_id,
cc.type_name,
cc.version_tag
FROM clusterconfig cc, clusterconfigmapping ccm
WHERE cc.config_id NOT IN (SELECT
scm.config_id
FROM serviceconfigmapping scm)
AND cc.type_name != 'cluster-env'
AND cc.type_name = ccm.type_name
AND cc.version_tag = ccm.version_tag);
DELETE FROM clusterconfigmapping
WHERE EXISTS
(SELECT 1 FROM orphaned_configs
WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
SELECT * FROM orphaned_configs;
DROP TABLE orphaned_configs;
{code}
I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
Diff: https://reviews.apache.org/r/58802/diff/2/
Changes: https://reviews.apache.org/r/58802/diff/1-2/
Testing
-------
mvn clean test
Also manual check on Derby
Thanks,
Dmitro Lisnichenko
Re: Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/#review173235
-----------------------------------------------------------
ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql
Lines 98 (patched)
<https://reviews.apache.org/r/58802/#comment246283>
use a TINYINT isntead?
- Jonathan Hurley
On April 27, 2017, 12:16 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58802/
> -----------------------------------------------------------
>
> (Updated April 27, 2017, 12:16 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-20875
> https://issues.apache.org/jira/browse/AMBARI-20875
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
>
> After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
>
> {noformat}
> 2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes
-site-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
> {noformat}
>
> These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
>
> A few options:
> - When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
>
> - Make the warning smarter somehow to ignore these since it's expected they are orphaned.
> -- Somehow determine the service which should own the config and see if it exists in the cluster?
> -- Add a new column to {{clusterconfig}} to mark it as deleted?
>
>
> To clean these warnings, we had to:
> {code}
> CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
> (SELECT
> cc.config_id,
> cc.type_name,
> cc.version_tag
> FROM clusterconfig cc, clusterconfigmapping ccm
> WHERE cc.config_id NOT IN (SELECT
> scm.config_id
> FROM serviceconfigmapping scm)
> AND cc.type_name != 'cluster-env'
> AND cc.type_name = ccm.type_name
> AND cc.version_tag = ccm.version_tag);
>
> DELETE FROM clusterconfigmapping
> WHERE EXISTS
> (SELECT 1 FROM orphaned_configs
> WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
>
> DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
>
> SELECT * FROM orphaned_configs;
>
> DROP TABLE orphaned_configs;
> {code}
>
> I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
> So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58802/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
> Also manual check on Derby
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 58802: Removing A Service Causes DB Verification To
Produce Warnings
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58802/#review173221
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
Lines 563 (patched)
<https://reviews.apache.org/r/58802/#comment246271>
Default in the entity is false (and non-null), so not sure you need Objects.equals() here.
ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql
Lines 77-78 (original), 77-79 (patched)
<https://reviews.apache.org/r/58802/#comment246272>
we usually use some type of SMALLINT for booleans instead of VARCHAR(1)'s See host_role_command.is_background
- Nate Cole
On April 27, 2017, 12:16 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58802/
> -----------------------------------------------------------
>
> (Updated April 27, 2017, 12:16 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-20875
> https://issues.apache.org/jira/browse/AMBARI-20875
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When removing a service, the configurations for that service are kept for historical purposes, but their various associations in the database are removed (specifically, the {{serviceconfigmapping}} relationships).
>
> After removing a service, the orphaned configurations now cause a warning to be displayed on Ambari Server startup:
>
> {noformat}
> 2017-04-06 17:15:24,003 WARN - You have config(s): ranger-storm-policymgr-ssl-version1467149286586,atlas-env-version1471883877194,falcon-env-version1467044148480,storm-site-version1467149286586,storm-site-version1474944944095,ranger-storm-plugin-properties-version1467149286586,hana_hadoop-env-version1476989318735,hana_hadoop-env-version1468951412523,hanaes-site-version1475773173499,hanaes-site-version1477639131416,atlas-env-version1471880496396,falcon-startup.properties-version1474944962583,ranger-storm-security-version1467149286586,falcon-env-version1474944962517,application-properties-version1471883877194,hanaes-site-version1468951412523,application-properties-version1471992143777,application-properties-version1471880496396,hana_hadoop-env-version1475790068354,hana_hadoop-env-version1477639131416,falcon-runtime.properties-version1467044148480,atlas-env-version1471992143777,hana_hadoop-env-version1475773173499,storm-env-version1467149286586,hanaes-site-version1475790068354,hanaes
-site-version1476902714170,atlas-env-version1471883827584,hana_hadoop-env-version1477695406433,hanaes-site-version1476989583427,falcon-log4j-version1,falcon-env-version1474944962457,hanaes-site-version1468959251565,falcon-client.properties-version1,atlas-env-version1471993347065,falcon-startup.properties-version1467044148480,storm-cluster-log4j-version1467149286586,hanaes-site-version1472285532383,hana_hadoop-env-version1477695089738,hana_hadoop-env-version1468959251565,hana_hadoop-env-version1476989821279,atlas-log4j-version1,storm-site-version1467612840864,storm-worker-log4j-version1467149286586,ranger-storm-audit-version1467149286586,application-properties-version1471993347065,application-properties-version1471883827584,hana_hadoop-env-version1477695579450 that is(are) not mapped (in serviceconfigmapping table) to any service!
> {noformat}
>
> These orphaned configurations have entries in both {{clusterconfig}} and {{clusterconfigmapping}} but are otherwise not references anywhere. They don't hurt anything, but do trigger this warning since we can't determine if they _should_ have mappings in {{serviceconfigmapping}}.
>
> A few options:
> - When removing a service, remove configurations as well, leaving no orphans. Some would argue that this is a bad move since re-adding the service later would allow you to see the old configurations. I do not believe this is true since the old configurations are never associated with the new service's {{serviceconfig}} or {{serviceconfigmapping}}.
>
> - Make the warning smarter somehow to ignore these since it's expected they are orphaned.
> -- Somehow determine the service which should own the config and see if it exists in the cluster?
> -- Add a new column to {{clusterconfig}} to mark it as deleted?
>
>
> To clean these warnings, we had to:
> {code}
> CREATE TEMPORARY TABLE IF NOT EXISTS orphaned_configs AS
> (SELECT
> cc.config_id,
> cc.type_name,
> cc.version_tag
> FROM clusterconfig cc, clusterconfigmapping ccm
> WHERE cc.config_id NOT IN (SELECT
> scm.config_id
> FROM serviceconfigmapping scm)
> AND cc.type_name != 'cluster-env'
> AND cc.type_name = ccm.type_name
> AND cc.version_tag = ccm.version_tag);
>
> DELETE FROM clusterconfigmapping
> WHERE EXISTS
> (SELECT 1 FROM orphaned_configs
> WHERE clusterconfigmapping.type_name = orphaned_configs.type_name AND clusterconfigmapping.version_tag = orphaned_configs.version_tag);
>
> DELETE FROM clusterconfig WHERE clusterconfig.config_id IN (SELECT config_id FROM orphaned_configs);
>
> SELECT * FROM orphaned_configs;
>
> DROP TABLE orphaned_configs;
> {code}
>
> I've considered advanced heuristics based on service metainfo with config dependencies, and service config mappings. But this approach may be unreliable when given inaccurate service metainfo between stack upgrades. Also, this approach is likely to delay server start on clusters with thousands of configs.
> So the solution Add a new column to clusterconfig to mark it as deleted. So warning will not be generated for such cluster configs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java c33c4e3815
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java b21aeff946
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java f96db609b7
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java e533850e14
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 1b13a84417
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java PRE-CREATION
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 43bf1def55
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 6def989c7f
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 1a0ab877b7
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5c6668cb78
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 37c69b2d11
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4fc6
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/58802/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
> Also manual check on Derby
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>