You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Laszlo Puskas <lp...@hortonworks.com> on 2017/03/03 08:16:12 UTC
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/
-----------------------------------------------------------
(Updated March 3, 2017, 8:16 a.m.)
Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
Changes
-------
Added check for another balancer running. Added message (to the backgroung operation popup) to explain how the command operates.
Bugs: AMBARI-20175
https://issues.apache.org/jira/browse/AMBARI-20175
Repository: ambari
Description (updated)
-------
The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
(NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
UPDATE:
The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f
Diff: https://reviews.apache.org/r/57040/diff/3/
Changes: https://reviews.apache.org/r/57040/diff/2-3/
Testing (updated)
-------
Manually tested on local unsecure cluster.
Manually tested on local secure cluster
Unit tests running.
Thanks,
Laszlo Puskas
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/#review167858
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On March 3, 2017, 12:30 p.m., Laszlo Puskas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57040/
> -----------------------------------------------------------
>
> (Updated March 3, 2017, 12:30 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
>
>
> Bugs: AMBARI-20175
> https://issues.apache.org/jira/browse/AMBARI-20175
>
>
> Repository: ambari
>
>
> Description
> -------
>
> The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
> As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
>
> (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
>
> UPDATE:
> The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py bd05a95
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py 08578bc
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py ec19dc4
>
>
> Diff: https://reviews.apache.org/r/57040/diff/4/
>
>
> Testing
> -------
>
> Manually tested on local unsecure cluster.
> Manually tested on local secure cluster
>
> Unit tests successfully ran.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/#review167817
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On March 3, 2017, 7:30 a.m., Laszlo Puskas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57040/
> -----------------------------------------------------------
>
> (Updated March 3, 2017, 7:30 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
>
>
> Bugs: AMBARI-20175
> https://issues.apache.org/jira/browse/AMBARI-20175
>
>
> Repository: ambari
>
>
> Description
> -------
>
> The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
> As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
>
> (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
>
> UPDATE:
> The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py bd05a95
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py 08578bc
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py ec19dc4
>
>
> Diff: https://reviews.apache.org/r/57040/diff/4/
>
>
> Testing
> -------
>
> Manually tested on local unsecure cluster.
> Manually tested on local secure cluster
>
> Unit tests successfully ran.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Laszlo Puskas <lp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/
-----------------------------------------------------------
(Updated March 3, 2017, 12:30 p.m.)
Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
Changes
-------
Updated to the patch against the trunk.
Bugs: AMBARI-20175
https://issues.apache.org/jira/browse/AMBARI-20175
Repository: ambari
Description
-------
The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
(NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
UPDATE:
The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py bd05a95
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_rebalance.py 1dc545e
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py 08578bc
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py ec19dc4
Diff: https://reviews.apache.org/r/57040/diff/4/
Changes: https://reviews.apache.org/r/57040/diff/3-4/
Testing
-------
Manually tested on local unsecure cluster.
Manually tested on local secure cluster
Unit tests successfully ran.
Thanks,
Laszlo Puskas
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/#review167815
-----------------------------------------------------------
Ship it!
Ship It!
- Sebastian Toader
On March 3, 2017, 9:52 a.m., Laszlo Puskas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57040/
> -----------------------------------------------------------
>
> (Updated March 3, 2017, 9:52 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
>
>
> Bugs: AMBARI-20175
> https://issues.apache.org/jira/browse/AMBARI-20175
>
>
> Repository: ambari
>
>
> Description
> -------
>
> The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
> As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
>
> (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
>
> UPDATE:
> The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f
>
>
> Diff: https://reviews.apache.org/r/57040/diff/3/
>
>
> Testing
> -------
>
> Manually tested on local unsecure cluster.
> Manually tested on local secure cluster
>
> Unit tests successfully ran.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/#review167814
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
Lines 349 (patched)
<https://reviews.apache.org/r/57040/#comment239794>
Maybe worth adding to the message that in case of very large clusters this operation may take hours or days to complete.
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
Lines 351-353 (patched)
<https://reviews.apache.org/r/57040/#comment239793>
Wouldn't be enough just writing out
"There is already a balancer running. Note that the process may take a long time to finish (hours, even days on very large clusters)."
- Sebastian Toader
On March 3, 2017, 9:52 a.m., Laszlo Puskas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57040/
> -----------------------------------------------------------
>
> (Updated March 3, 2017, 9:52 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
>
>
> Bugs: AMBARI-20175
> https://issues.apache.org/jira/browse/AMBARI-20175
>
>
> Repository: ambari
>
>
> Description
> -------
>
> The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
> As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
>
> (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
>
> UPDATE:
> The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f
>
>
> Diff: https://reviews.apache.org/r/57040/diff/3/
>
>
> Testing
> -------
>
> Manually tested on local unsecure cluster.
> Manually tested on local secure cluster
>
> Unit tests successfully ran.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 57040: Rebalance HDFS operation returns after the
command is issued
Posted by Laszlo Puskas <lp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57040/
-----------------------------------------------------------
(Updated March 3, 2017, 8:52 a.m.)
Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, and Sebastian Toader.
Bugs: AMBARI-20175
https://issues.apache.org/jira/browse/AMBARI-20175
Repository: ambari
Description
-------
The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out.
As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish.
(NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation)
UPDATE:
The operation is only triggered if there's no other rebalance operation running. (This is done by checking the existence of the /system/balancer.id on hdfs As per the input from hdfs devs, this check may be misleading if a balancer process is killed by an admin, as the file can remain there. - To overcome this added a message to explain this behavior)
Diffs
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_rebalance.py 1dc545e
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f
Diff: https://reviews.apache.org/r/57040/diff/3/
Testing (updated)
-------
Manually tested on local unsecure cluster.
Manually tested on local secure cluster
Unit tests successfully ran.
Thanks,
Laszlo Puskas