You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Tim Thorpe <tt...@ca.ibm.com> on 2016/12/19 21:24:19 UTC
Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
Bugs: AMBARI-19241
https://issues.apache.org/jira/browse/AMBARI-19241
Repository: ambari
Description
-------
Currently the hdfs_resources.py supports
1) create files/directories in HDFS
2) delete files/directories in HDFS
3) upload files/directories to HDFS
We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
Diff: https://reviews.apache.org/r/54874/diff/
Testing
-------
Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
Thanks,
Tim Thorpe
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Tim Thorpe <tt...@ca.ibm.com>.
> On Jan. 10, 2017, 9:42 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py, line 168
> > <https://reviews.apache.org/r/54874/diff/1/?file=1589085#file1589085line168>
> >
> > Does thie curl command work when Kerberos / SSL is being used? Should you instead use curl_krb_request?
>
> Tim Thorpe wrote:
> This is the same curl command which has been used for creating files in HDFS, upload, querying the file status, etc...
> I would assume this has been tested many times with Kerberos and SSL. I don't see why changing the URL to be an OPEN operation would impact that but I can test Kerberos/SSL.
>
> Jonathan Hurley wrote:
> Great - it might not be an issue, but I thought I'd check. If you can confirm it works with Kerberos, then you can drop the issue.
under def action_execute(self, main_resource), it runs the kinit:
if security_enabled:
main_resource.kinit()
This avoids doing a kinit for each call to HDFS. The HDFSResource is gathering a queue of actions to be performed to avoid all the expensive operations of opening connections. Using curl_krb_request would defeat the point.
The actual curl command handles the rest of the security in the run_command function (where the curl command you listed is found):
if self.security_enabled:
cmd += ["--negotiate", "-u", ":"]
if self.is_https_enabled:
cmd += ["-k"]
- Tim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review161127
-----------------------------------------------------------
On Dec. 19, 2016, 9:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Tim Thorpe <tt...@ca.ibm.com>.
> On Jan. 10, 2017, 9:42 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py, line 168
> > <https://reviews.apache.org/r/54874/diff/1/?file=1589085#file1589085line168>
> >
> > Does thie curl command work when Kerberos / SSL is being used? Should you instead use curl_krb_request?
This is the same curl command which has been used for creating files in HDFS, upload, querying the file status, etc...
I would assume this has been tested many times with Kerberos and SSL. I don't see why changing the URL to be an OPEN operation would impact that but I can test Kerberos/SSL.
- Tim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review161127
-----------------------------------------------------------
On Dec. 19, 2016, 9:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Jan. 10, 2017, 4:42 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py, line 168
> > <https://reviews.apache.org/r/54874/diff/1/?file=1589085#file1589085line168>
> >
> > Does thie curl command work when Kerberos / SSL is being used? Should you instead use curl_krb_request?
>
> Tim Thorpe wrote:
> This is the same curl command which has been used for creating files in HDFS, upload, querying the file status, etc...
> I would assume this has been tested many times with Kerberos and SSL. I don't see why changing the URL to be an OPEN operation would impact that but I can test Kerberos/SSL.
>
> Jonathan Hurley wrote:
> Great - it might not be an issue, but I thought I'd check. If you can confirm it works with Kerberos, then you can drop the issue.
>
> Tim Thorpe wrote:
> under def action_execute(self, main_resource), it runs the kinit:
> if security_enabled:
> main_resource.kinit()
> This avoids doing a kinit for each call to HDFS. The HDFSResource is gathering a queue of actions to be performed to avoid all the expensive operations of opening connections. Using curl_krb_request would defeat the point.
>
> The actual curl command handles the rest of the security in the run_command function (where the curl command you listed is found):
> if self.security_enabled:
> cmd += ["--negotiate", "-u", ":"]
> if self.is_https_enabled:
> cmd += ["-k"]
Yep - the -k and --negotiate params were what I didn't see. Thanks!
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review161127
-----------------------------------------------------------
On Dec. 19, 2016, 4:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 4:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Jan. 10, 2017, 4:42 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py, line 168
> > <https://reviews.apache.org/r/54874/diff/1/?file=1589085#file1589085line168>
> >
> > Does thie curl command work when Kerberos / SSL is being used? Should you instead use curl_krb_request?
>
> Tim Thorpe wrote:
> This is the same curl command which has been used for creating files in HDFS, upload, querying the file status, etc...
> I would assume this has been tested many times with Kerberos and SSL. I don't see why changing the URL to be an OPEN operation would impact that but I can test Kerberos/SSL.
Great - it might not be an issue, but I thought I'd check. If you can confirm it works with Kerberos, then you can drop the issue.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review161127
-----------------------------------------------------------
On Dec. 19, 2016, 4:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 4:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review161127
-----------------------------------------------------------
Fix it, then Ship it!
ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py (line 168)
<https://reviews.apache.org/r/54874/#comment232365>
Does thie curl command work when Kerberos / SSL is being used? Should you instead use curl_krb_request?
- Jonathan Hurley
On Dec. 19, 2016, 4:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 4:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 54874: AMBARI-19241 - Ambari python scripts should
support hdfs download
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54874/#review159720
-----------------------------------------------------------
Ship it!
Ship It!
- Di Li
On Dec. 19, 2016, 9:24 p.m., Tim Thorpe wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54874/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Di Li, Jonathan Hurley, Jayush Luniya, and Sumit Mohanty.
>
>
> Bugs: AMBARI-19241
> https://issues.apache.org/jira/browse/AMBARI-19241
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently the hdfs_resources.py supports
> 1) create files/directories in HDFS
> 2) delete files/directories in HDFS
> 3) upload files/directories to HDFS
>
> We should also support download of files/directories from HDFS. This will help particularly in cloud environments where the users of the cluster don't necessarily have write access to the local file system.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/download_from_hdfs.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py f1aa3e1
> ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py 5761fd6
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Resource.java da5a706
> contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java e4656c7
>
> Diff: https://reviews.apache.org/r/54874/diff/
>
>
> Testing
> -------
>
> Manual testing with and without web hdfs. Tested download files and directories both when they existed locally and when they didn't.
>
>
> Thanks,
>
> Tim Thorpe
>
>