You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/09/26 23:03:59 UTC
Review Request 38788: RU - HDFS_Client restart and hdp-select causes
dfs_data_dir_mount.hist to be lost
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py dd0dca4
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100819
-----------------------------------------------------------
Can you update dfs_datanode_helper.py as well?
ambari-common/src/main/python/resource_management/libraries/functions/dfs_datanode_helper.py: This function relies on the dfs.datanode.data.dir.mount.file parameter to parse a file that contains
ambari-common/src/main/python/resource_management/libraries/functions/dfs_datanode_helper.py: Logger.warning("DataNode is allowed to create any data directory since dfs.datanode.data.dir.mount.file property is null.")
ambari-common/src/main/python/resource_management/libraries/functions/dfs_datanode_helper.py: Logger.warning("DataNode is allowed to create any data directory since dfs.datanode.data.dir.mount.file property has file %s and it does not exist." % params.data_dir_mount_file)
- Jayush Luniya
On Sept. 28, 2015, 5:29 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2015, 5:29 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py 2912406
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3160af2
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py fb0a4db
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
> ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
> ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
> ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
> ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
> ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
> ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
> ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
> ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
> ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
> ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
> ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
> ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
> ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
> ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
> ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
> ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
> ambari-web/app/assets/data/configurations/config_versions.json 603d33e
> ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
> ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
> ambari-web/app/data/BIGTOP/site_properties.js e71cab4
> ambari-web/app/data/HDP2/site_properties.js 03ffd31
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
(Updated Sept. 28, 2015, 5:31 p.m.)
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py 2912406
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3160af2
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py fb0a4db
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
ambari-web/app/assets/data/configurations/config_versions.json 603d33e
ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
ambari-web/app/data/BIGTOP/site_properties.js e71cab4
ambari-web/app/data/HDP2/site_properties.js 03ffd31
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100791
-----------------------------------------------------------
ambari-agent/src/main/python/ambari_agent/Controller.py (line 102)
<https://reviews.apache.org/r/38788/#comment158038>
Lets add a try: ... except: to ignore any failures during agent start.
- Sumit Mohanty
On Sept. 28, 2015, 5:29 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2015, 5:29 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py 2912406
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3160af2
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py fb0a4db
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
> ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
> ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
> ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
> ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
> ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
> ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
> ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
> ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
> ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
> ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
> ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
> ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
> ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
> ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
> ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
> ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
> ambari-web/app/assets/data/configurations/config_versions.json 603d33e
> ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
> ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
> ambari-web/app/data/BIGTOP/site_properties.js e71cab4
> ambari-web/app/data/HDP2/site_properties.js 03ffd31
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100792
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Sept. 28, 2015, 5:29 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2015, 5:29 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py 2912406
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3160af2
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py fb0a4db
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
> ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
> ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
> ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
> ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
> ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
> ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
> ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
> ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
> ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
> ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
> ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
> ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
> ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
> ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
> ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
> ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
> ambari-web/app/assets/data/configurations/config_versions.json 603d33e
> ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
> ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
> ambari-web/app/data/BIGTOP/site_properties.js e71cab4
> ambari-web/app/data/HDP2/site_properties.js 03ffd31
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
(Updated Sept. 28, 2015, 5:29 a.m.)
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Changes
-------
Removed config for dfs.datanode.data.dir.mount.file. Still need to test it.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py 2912406
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3160af2
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py fb0a4db
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
ambari-web/app/assets/data/configurations/config_versions.json 603d33e
ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
ambari-web/app/data/BIGTOP/site_properties.js e71cab4
ambari-web/app/data/HDP2/site_properties.js 03ffd31
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Sumit Mohanty <sm...@hortonworks.com>.
> On Sept. 27, 2015, 8:39 a.m., Jayush Luniya wrote:
> > ambari-agent/src/main/python/ambari_agent/Controller.py, line 450
> > <https://reviews.apache.org/r/38788/diff/4/?file=1085372#file1085372line450>
> >
> > Since dfs.datanode.data.dir.mount.file is a stack config property how will we handle the case where the path was set by user to a different location before ambari upgrade?
How about not letting user set this value anymore? Its in /var/lib/ambari-agent/data and that need not be changed.
We can look for the file in two places if there is a possibility that users have change the location. Looking at the nature of the config it seems less likely that users will change the file location in hadoop-env.
- Sumit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100773
-----------------------------------------------------------
On Sept. 27, 2015, 7:20 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 27, 2015, 7:20 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
> ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
> ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
> ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
> ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
> ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
> ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
> ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
> ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
> ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
> ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
> ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
> ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
> ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
> ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
> ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
> ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
> ambari-web/app/assets/data/configurations/config_versions.json 603d33e
> ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
> ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
> ambari-web/app/data/BIGTOP/site_properties.js e71cab4
> ambari-web/app/data/HDP2/site_properties.js 03ffd31
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100773
-----------------------------------------------------------
ambari-agent/src/main/python/ambari_agent/Controller.py (line 450)
<https://reviews.apache.org/r/38788/#comment158024>
Since dfs.datanode.data.dir.mount.file is a stack config property how will we handle the case where the path was set by user to a different location before ambari upgrade?
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java (line 388)
<https://reviews.apache.org/r/38788/#comment158025>
Similar case here: Should we change the path if the user has set it to a different location?
- Jayush Luniya
On Sept. 27, 2015, 7:20 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 27, 2015, 7:20 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
> ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
> ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
> ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
> ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
> ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
> ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
> ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
> ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
> ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
> ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
> ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
> ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
> ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
> ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
> ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
> ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
> ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
> ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
> ambari-web/app/assets/data/configurations/config_versions.json 603d33e
> ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
> ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
> ambari-web/app/data/BIGTOP/site_properties.js e71cab4
> ambari-web/app/data/HDP2/site_properties.js 03ffd31
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
(Updated Sept. 27, 2015, 7:20 a.m.)
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py 34ec8cd
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
ambari-web/app/assets/data/configurations/config_versions.json 603d33e
ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
ambari-web/app/data/BIGTOP/site_properties.js e71cab4
ambari-web/app/data/HDP2/site_properties.js 03ffd31
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
(Updated Sept. 27, 2015, 3:05 a.m.)
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Changes
-------
Used mv command, and updated defaults.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml 4cb2274
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml c6dd202
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml b5451d8
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py 4406231
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0
ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9
ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085
ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e
ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b
ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728
ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json b2fd6e8
ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8
ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9
ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259
ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df
ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6
ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6
ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918
ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9
ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11
ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec
ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8
ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5
ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487
ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621
ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b
ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c
ambari-web/app/assets/data/configurations/config_versions.json 603d33e
ambari-web/app/assets/data/configurations/service_versions.json c5eb46c
ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656
ambari-web/app/data/BIGTOP/site_properties.js e71cab4
ambari-web/app/data/HDP2/site_properties.js 03ffd31
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Mahadev Konar <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100769
-----------------------------------------------------------
Is there a default for hadoop-env we need to change?
- Mahadev Konar
On Sept. 27, 2015, 2:36 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 27, 2015, 2:36 a.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------
(Updated Sept. 27, 2015, 2:36 a.m.)
Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
Changes
-------
Switched to a different strategy, move the file to a static location.
Bugs: AMBARI-13251
https://issues.apache.org/jira/browse/AMBARI-13251
Repository: ambari
Description
-------
During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
it means that the file must be copied to the new location.
Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/Controller.py 8746172
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab
Diff: https://reviews.apache.org/r/38788/diff/
Testing
-------
Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 38788: RU - HDFS_Client restart and hdp-select
causes dfs_data_dir_mount.hist to be lost
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100760
-----------------------------------------------------------
Ship it!
Ship It!
- Jayush Luniya
On Sept. 26, 2015, 9:03 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
>
> (Updated Sept. 26, 2015, 9:03 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
>
>
> Bugs: AMBARI-13251
> https://issues.apache.org/jira/browse/AMBARI-13251
>
>
> Repository: ambari
>
>
> Description
> -------
>
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/ from the old version to the new version.
>
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf
> it means that the file must be copied to the new location.
>
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of those clients it is the first one in the Upgrade Pack.
>
> During a Rolling Downgrade, the file should already exist in the previous version, and the file is only written to when DataNode is restarted, so we are ok.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py dd0dca4
>
> Diff: https://reviews.apache.org/r/38788/diff/
>
>
> Testing
> -------
>
> Tested the alert during RU, and the mount history file was copied to the new location, so no warnings were generated.
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>