You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/02/05 03:16:29 UTC

Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/
-----------------------------------------------------------

Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.

Bugs: AMBARI-9485
https://issues.apache.org/jira/browse/AMBARI-9485

Repository: ambari

Description
-------

Add more debugging information for HDFS prepare and finalize steps.

Prepare:
1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
2. With non-HA setup, the current code requires the NameNode in safemode when start RU.

The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.

We need to log what the original state is.

Finalize:
This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch

```
[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
QUERY rolling upgrade ...
Proceed with rolling upgrade:
Block Pool ID: BP-819970438-192.168.64.104-1423091248521
Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
Finalize Time: <NOT FINALIZED>

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
FINALIZE rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
QUERY rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.
```

This means that the finalize did happen.
We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.

Diffs
-----

ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c

Diff: https://reviews.apache.org/r/30648/diff/

Testing
-------

Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.

Unit tests are in progress.

Thanks,

Alejandro Fernandez

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.


> On Feb. 5, 2015, 7:01 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py, line 29
> > <https://reviews.apache.org/r/30648/diff/1/?file=849486#file849486line29>
> >
> >     Class name in all caps?

Will change it to SafeMode.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71270
-----------------------------------------------------------


On Feb. 5, 2015, 8 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 8 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py ef0991e 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71270
-----------------------------------------------------------

Ship it!



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
<https://reviews.apache.org/r/30648/#comment116944>

    Class name in all caps?


- Jonathan Hurley


On Feb. 4, 2015, 9:47 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2015, 9:47 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Tom Beerbower <tb...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71375
-----------------------------------------------------------

Ship it!


Ship It!

- Tom Beerbower


On Feb. 5, 2015, 8:04 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 8:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Nate Cole <nc...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71370
-----------------------------------------------------------

Ship it!


Ship It!

- Nate Cole


On Feb. 5, 2015, 3:04 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 3:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71430
-----------------------------------------------------------

Ship it!


Ship It!

- Jonathan Hurley


On Feb. 5, 2015, 3:04 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 3:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.

(Updated Feb. 5, 2015, 8:04 p.m.)

Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.

Bugs: AMBARI-9485
https://issues.apache.org/jira/browse/AMBARI-9485

Repository: ambari

Description
-------

Add more debugging information for HDFS prepare and finalize steps.

Prepare:
1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
2. With non-HA setup, the current code requires the NameNode in safemode when start RU.

We need to log what the original state is.

Finalize:
This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
FINALIZE rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
QUERY rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.
```

This means that the finalize did happen.
We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.

Diffs (updated)
-----

ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c

Diff: https://reviews.apache.org/r/30648/diff/

Testing
-------

Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.

Unit tests passed.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29:57.240s
[INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
[INFO] Final Memory: 60M/496M
[INFO] ------------------------------------------------------------------------

Thanks,

Alejandro Fernandez

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71291
-----------------------------------------------------------



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py
<https://reviews.apache.org/r/30648/#comment116962>

    Unrelated change when Journalnode is not present.


- Alejandro Fernandez


On Feb. 5, 2015, 8 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 8 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py ef0991e 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.

(Updated Feb. 5, 2015, 8 p.m.)

Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.

Changes
-------

Addressed Nate's and Jonathan's comments.

Bugs: AMBARI-9485
https://issues.apache.org/jira/browse/AMBARI-9485

Repository: ambari

Description
-------

Add more debugging information for HDFS prepare and finalize steps.

Prepare:
1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
2. With non-HA setup, the current code requires the NameNode in safemode when start RU.

We need to log what the original state is.

Finalize:
This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
FINALIZE rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
QUERY rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.
```

This means that the finalize did happen.
We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.

Diffs (updated)
-----

ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py ef0991e

Diff: https://reviews.apache.org/r/30648/diff/

Testing
-------

Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.

Unit tests passed.

Thanks,

Alejandro Fernandez

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.


> On Feb. 5, 2015, 12:15 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py, lines 52-55
> > <https://reviews.apache.org/r/30648/diff/1/?file=849486#file849486line52>
> >
> >     Any kinit concerns for these commands?

kinit is the first thing that happens in namenode_upgrade.py prepare_rolling_upgrade().


> On Feb. 5, 2015, 12:15 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py, line 60
> > <https://reviews.apache.org/r/30648/diff/1/?file=849486#file849486line60>
> >
> >     Should you be using the same grep pattern as above?  This regex looks like it's only for HA?

Good catch, will fix it.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71200
-----------------------------------------------------------


On Feb. 5, 2015, 2:47 a.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2015, 2:47 a.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Nate Cole <nc...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30648/#review71200
-----------------------------------------------------------



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
<https://reviews.apache.org/r/30648/#comment116874>

    Any kinit concerns for these commands?



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
<https://reviews.apache.org/r/30648/#comment116875>

    Should you be using the same grep pattern as above?  This regex looks like it's only for HA?


- Nate Cole


On Feb. 4, 2015, 9:47 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30648/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2015, 9:47 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.
> 
> 
> Bugs: AMBARI-9485
>     https://issues.apache.org/jira/browse/AMBARI-9485
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add more debugging information for HDFS prepare and finalize steps.
> 
> Prepare:
> 1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
> 2. With non-HA setup, the current code requires the NameNode in safemode when start RU.
> 
> The reason is that for non-HA setup, we need to do an extra checkpoint in the single NN (by calling saveNamespace), which requires the NN to be in safemode (to avoid namespace modification during the checkpoint). But for HA-setup, the checkpoint is done by the standby NN, thus we do not need to put the active NN into the safemode and in this way we can avoid service downtime.
> 
> We need to log what the original state is.
> 
> Finalize:
> This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch
> 
> ```
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> Proceed with rolling upgrade:
>   Block Pool ID: BP-819970438-192.168.64.104-1423091248521
>      Start Time: Thu Feb 05 01:15:16 UTC 2015 (=1423098916798)
>   Finalize Time: <NOT FINALIZED>
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
> FINALIZE rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> 
> [root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
> QUERY rolling upgrade ...
> There is no rolling upgrade in progress or rolling upgrade has already been finalized.
> ```
> 
> This means that the finalize did happen.
> We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c 
> 
> Diff: https://reviews.apache.org/r/30648/diff/
> 
> 
> Testing
> -------
> 
> Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.
> 
> Unit tests passed.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 29:57.240s
> [INFO] Finished at: Wed Feb 04 18:47:14 PST 2015
> [INFO] Final Memory: 60M/496M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 30648: RU - Add more debugging information to HDFS prepare and finalize steps

Posted by Alejandro Fernandez <af...@hortonworks.com>.

(Updated Feb. 5, 2015, 2:47 a.m.)

Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Tom Beerbower, and Yurii Shylov.

Changes
-------

Unit tests passed.

Bugs: AMBARI-9485
https://issues.apache.org/jira/browse/AMBARI-9485

Repository: ambari

Description
-------

Add more debugging information for HDFS prepare and finalize steps.

Prepare:
1. With HA setup, the current code requires the NameNode NOT in safemode when starting RU.
2. With non-HA setup, the current code requires the NameNode in safemode when start RU.

We need to log what the original state is.

Finalize:
This has a confusing message related to https://issues.apache.org/jira/secure/attachment/12679263/HDFS-7340.001.patch

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade finalize'
FINALIZE rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.

[root@c6404 ~]# su - hdfs -c 'hdfs dfsadmin -rollingUpgrade query'
QUERY rolling upgrade ...
There is no rolling upgrade in progress or rolling upgrade has already been finalized.
```

This means that the finalize did happen.
We should run another "hdfs dfsadmin -rollingUpgrade query" after finalize to ensure this happened.

Diffs
-----

ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py e11546c

Diff: https://reviews.apache.org/r/30648/diff/

Testing (updated)
-------

Installed a 3-node cluster with HDFS and ZK, and Namenode HA, then verified that Namenode prepare showed the additional output, and that HDFS finalize made it clear that it did happen.

Unit tests passed.

Thanks,

Alejandro Fernandez