You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/05/30 18:21:50 UTC
Review Request 34846: Ooozie start takes too long
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34846/
-----------------------------------------------------------
Review request for Ambari and Dmytro Sen.
Bugs: AMBARI-11558
https://issues.apache.org/jira/browse/AMBARI-11558
Repository: ambari
Description
-------
Currently Oozie Start task takes ~3 minutes. Which includes some time-
expensive actions like:
1\. Extracting a big tar archive
2\. Execute prepare-war fom it
3\. Checking for hdfs directory via hadoop binary.
1 and 2 can be avoid on the non-first starts, by saving checksum of archive
which was extracted, and not re-extracting it unless the checksum changed, or
the unextracted folder is gone.
3 can benefit from using fast WebHDFS calls.
**As a result for me for non-first Oozie start this reduced time of start from 180 seconds to 13-30 seconds.**
Diffs
-----
ambari-common/src/main/python/resource_management/core/logger.py c060753
ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 25f0cd5
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py c773fca
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py ef721ba
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 6963410
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 284f392
ambari-server/src/test/python/stacks/utils/RMFTestCase.py 2f608c4
Diff: https://reviews.apache.org/r/34846/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 34846: Ooozie start takes too long
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34846/#review85869
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Sen
On Май 30, 2015, 4:21 п.п., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34846/
> -----------------------------------------------------------
>
> (Updated Май 30, 2015, 4:21 п.п.)
>
>
> Review request for Ambari and Dmytro Sen.
>
>
> Bugs: AMBARI-11558
> https://issues.apache.org/jira/browse/AMBARI-11558
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently Oozie Start task takes ~3 minutes. Which includes some time-
> expensive actions like:
> 1\. Extracting a big tar archive
> 2\. Execute prepare-war fom it
> 3\. Checking for hdfs directory via hadoop binary.
>
> 1 and 2 can be avoid on the non-first starts, by saving checksum of archive
> which was extracted, and not re-extracting it unless the checksum changed, or
> the unextracted folder is gone.
>
> 3 can benefit from using fast WebHDFS calls.
>
> **As a result for me for non-first Oozie start this reduced time of start from 180 seconds to 13-30 seconds.**
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/logger.py c060753
> ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 25f0cd5
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py c773fca
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py ef721ba
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 6963410
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 284f392
> ambari-server/src/test/python/stacks/utils/RMFTestCase.py 2f608c4
>
> Diff: https://reviews.apache.org/r/34846/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>