You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2014/02/27 01:21:51 UTC
Review Request 18548: Move copyFromLocal to resource management package
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18548/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit Mohanty.
Bugs: AMBARI-4767
https://issues.apache.org/jira/browse/AMBARI-4767
Repository: ambari
Description
-------
Currently it seems the python library does not deal with any explicit hadoop commands like copyFromLocal.
We have a place for all the custom common resources, So if you have resource which is used in more than one service it worth to do that and add it to
resource_management/libraries/providers/
Diffs
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 8d557f2
ambari-agent/src/main/python/resource_management/libraries/providers/copy_from_local.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py a072455
ambari-agent/src/main/python/resource_management/libraries/resources/copy_from_local.py PRE-CREATION
ambari-agent/src/test/python/resource_management/TestCopyFromLocal.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 5210d3a
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/WEBHCAT/package/scripts/webhcat.py 7aa8521
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py cc480dc
ambari-server/src/test/python/stacks/2.0.6/WEBHCAT/test_webhcat_server.py 23c7e27
Diff: https://reviews.apache.org/r/18548/diff/
Testing
-------
Thanks,
Sid Wagle
Re: Review Request 18548: Move copyFromLocal to resource management package
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18548/#review35658
-----------------------------------------------------------
Ship it!
Ship It!
- Andrew Onischuk
On Feb. 27, 2014, 6:23 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18548/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 6:23 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-4767
> https://issues.apache.org/jira/browse/AMBARI-4767
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently it seems the python library does not deal with any explicit hadoop commands like copyFromLocal.
>
> We have a place for all the custom common resources, So if you have resource which is used in more than one service it worth to do that and add it to
> resource_management/libraries/providers/
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 8d557f2
> ambari-agent/src/main/python/resource_management/libraries/providers/copy_from_local.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py a072455
> ambari-agent/src/main/python/resource_management/libraries/resources/copy_from_local.py PRE-CREATION
> ambari-agent/src/test/python/resource_management/TestCopyFromLocal.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 5210d3a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/WEBHCAT/package/scripts/webhcat.py 7aa8521
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py cc480dc
> ambari-server/src/test/python/stacks/2.0.6/WEBHCAT/test_webhcat_server.py 23c7e27
>
> Diff: https://reviews.apache.org/r/18548/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 18548: Move copyFromLocal to resource management package
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18548/
-----------------------------------------------------------
(Updated Feb. 27, 2014, 6:23 p.m.)
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit Mohanty.
Changes
-------
Incorporated comments.
Bugs: AMBARI-4767
https://issues.apache.org/jira/browse/AMBARI-4767
Repository: ambari
Description
-------
Currently it seems the python library does not deal with any explicit hadoop commands like copyFromLocal.
We have a place for all the custom common resources, So if you have resource which is used in more than one service it worth to do that and add it to
resource_management/libraries/providers/
Diffs (updated)
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 8d557f2
ambari-agent/src/main/python/resource_management/libraries/providers/copy_from_local.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py a072455
ambari-agent/src/main/python/resource_management/libraries/resources/copy_from_local.py PRE-CREATION
ambari-agent/src/test/python/resource_management/TestCopyFromLocal.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 5210d3a
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/WEBHCAT/package/scripts/webhcat.py 7aa8521
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py cc480dc
ambari-server/src/test/python/stacks/2.0.6/WEBHCAT/test_webhcat_server.py 23c7e27
Diff: https://reviews.apache.org/r/18548/diff/
Testing
-------
Thanks,
Sid Wagle
Re: Review Request 18548: Move copyFromLocal to resource management package
Posted by Sid Wagle <sw...@hortonworks.com>.
> On Feb. 27, 2014, 4:06 p.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py, line 118
> > <https://reviews.apache.org/r/18548/diff/1/?file=505201#file505201line118>
> >
> > Sid, maybe let's not use total ignoring, since when we have some issues it's hard to debug on command failure. Let's write proper shell cmd to grep out "already exists" messages and give 0?
> >
> > Also I can't understand why do we need this stub_path, can you add a comment or rename it?
- Actually, the ignore is passed in asis to ExecuteHadoop, can we do that higher up the call chain ? If yes, I can open a separate Jira for that.
- The stub is need when copying multiple files to a hdfs directory so copying say /x/y/prefix*.extension, we need to create the target directory and the copy from local, the unless command in this case will look like "hadoop fs -ls /apps/tez/lib/*.jar", which will work too. I will make this change a part of AMBARI-4862.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18548/#review35628
-----------------------------------------------------------
On Feb. 27, 2014, 12:21 a.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18548/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 12:21 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-4767
> https://issues.apache.org/jira/browse/AMBARI-4767
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently it seems the python library does not deal with any explicit hadoop commands like copyFromLocal.
>
> We have a place for all the custom common resources, So if you have resource which is used in more than one service it worth to do that and add it to
> resource_management/libraries/providers/
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 8d557f2
> ambari-agent/src/main/python/resource_management/libraries/providers/copy_from_local.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py a072455
> ambari-agent/src/main/python/resource_management/libraries/resources/copy_from_local.py PRE-CREATION
> ambari-agent/src/test/python/resource_management/TestCopyFromLocal.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 5210d3a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/WEBHCAT/package/scripts/webhcat.py 7aa8521
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py cc480dc
> ambari-server/src/test/python/stacks/2.0.6/WEBHCAT/test_webhcat_server.py 23c7e27
>
> Diff: https://reviews.apache.org/r/18548/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 18548: Move copyFromLocal to resource management package
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18548/#review35628
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
<https://reviews.apache.org/r/18548/#comment66262>
Sid, maybe let's not use total ignoring, since when we have some issues it's hard to debug on command failure. Let's write proper shell cmd to grep out "already exists" messages and give 0?
Also I can't understand why do we need this stub_path, can you add a comment or rename it?
- Andrew Onischuk
On Feb. 27, 2014, 12:21 a.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18548/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 12:21 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-4767
> https://issues.apache.org/jira/browse/AMBARI-4767
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently it seems the python library does not deal with any explicit hadoop commands like copyFromLocal.
>
> We have a place for all the custom common resources, So if you have resource which is used in more than one service it worth to do that and add it to
> resource_management/libraries/providers/
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 8d557f2
> ambari-agent/src/main/python/resource_management/libraries/providers/copy_from_local.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py a072455
> ambari-agent/src/main/python/resource_management/libraries/resources/copy_from_local.py PRE-CREATION
> ambari-agent/src/test/python/resource_management/TestCopyFromLocal.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py 5210d3a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/WEBHCAT/package/scripts/webhcat.py 7aa8521
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py cc480dc
> ambari-server/src/test/python/stacks/2.0.6/WEBHCAT/test_webhcat_server.py 23c7e27
>
> Diff: https://reviews.apache.org/r/18548/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sid Wagle
>
>